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Abstract 

Unlike  aircrew  directly  associated  with  acquisitions,  line  operators  are  not  fully 
engaged  in  the  methods  to  push  materiel — hardware  or  software — change  requests  up  the 
chain,  to  a  decision  maker,  and  then  to  the  engineers.  The  Air  Force  trains  these  end 
users  to  logically  apply  expert  systems  knowledge  to  execute  the  mission  but  has  not 
fully  leveraged  this  resource  for  properly  identifying  and  correcting  operational  shortfalls 
in  an  aircraft’s  design.  Focusing  on  the  Remotely  Piloted  Aircraft  (RPA)  community,  the 
research  goal  is  to  determine  if  the  Air  Force  should  establish  a  formal  program  for 
collecting  and  prioritizing  unsolicited  user  change  requests  from  operators,  and  if  so,  how 
should  the  process  be  implemented  and  what  characteristics  should  the  system  possess. 
This  Delphi  study  sought  consensus  from  a  panel  of  MQ-1  and  MQ-9  expert  operators  on 
desired  characteristics  and  basic  architecture.  The  analysis  revealed  that  the  deficiency 
reporting  program,  traditionally  focused  on  Test  &  Evaluation  squadrons,  meets  many  of 
the  desired  characteristics  but  could  be  improved  to  meet  all  of  them.  Additionally, 
cockpit  development  could  improve  through  supplementing  the  already  established 
Cockpit  Working  Groups  with  a  commercially  developed  tool  with  many  of  the  desired 
characteristics. 
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COLLECTING  UNSOLICTED  USER-GENERATED  CHANGE  REQUESTS 


I.  Introduction 

General  Issue 

End  users  are  often  forced  to  support  their  commander’s  goals  with  less-than- 
ideal  equipment  due  to  requirements  uncertainty  coupled  with  ever-changing  mission 
requirements.  Potential  improvement  ideas,  especially  for  complex  solutions  to  complex 
problems,  could  be  constantly  elicited  from  the  people  who  are  most  familiar  with  the 
current  system’s  limitations:  the  end  users  also  known  as  line  operators.  A  continuous 
flow  of  end  user  feedback  is  a  potential  way  to  improve  system  perfonnance  through 
capturing  evolving  requirements. 

Problem  Statement 

Unlike  aircrew  directly  associated  with  acquisitions,  i.e.  assigned  to  the  System 
Program  Office  (SPO)  or  flying  with  an  operational  or  developmental  test  squadron,  line 
aircrew  are  not  fully  engaged  in  the  methods  to  push  hardware  or  software  change 
requests  up  the  chain  and  to  the  engineers.  The  primary  job  of  a  line  airman  is  to  fly: 
combat  sorties,  combat-support  sorties  or  basic  mission  proficiency  sorties  in  preparation 
for  combat  or  combat  support.  Collectively,  line  aircrew  are  the  most  knowledgeable 
resource  for  knowing  how  their  particular  weapon  system  is  used  operationally.  This 
study  specifically  studies  the  RPA  community — namely  the  aircrew  associated  with  the 
MQ- 1  Predator  and  MQ-9  Reaper. 

Unfortunately,  the  Air  Force  has  developed,  but  left  partially  untapped,  this  huge 
resource  for  properly  identifying  and  correcting  operational  shortfalls  in  the  design  of 
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specific  equipment.  The  Air  Force  favors  technical  degrees  for  aircrew  and  then  trains 
the  crewmember  to  logically  apply  expert  systems  knowledge  to  execute  a  mission.  This 
training  and  background  allows  line  aircrew  to  identify  design  errors  of  their  weapon 
system  in  a  variety  of  different  applications.  Unfortunately  the  deliberations  of  this 
valuable  resource  are  often  left  to  ferment  at  the  lowest  levels  instead  of  being  distilled 
into  more  capable,  more  effective,  and  more  efficient  equipment. 

While  the  Air  Force  does  have  processes  that  allow  user-initiated  feedback,  none 
of  the  current  programs  are  oriented  towards  collecting  software  or  hardware  changes 
from  individual  line  operators.  Most  aircrew  are  familiar  with  the  Fonn  847 — a  program 
for  changing  publications  such  as  operator  technical  orders  or  directive  policies. 
Mirroring  the  operator’s  Fonn  847,  the  maintenance  community  developed  the  AFTO 
Form  22  with  a  similar  functional  effect  for  maintenance  technical  orders.  Less  known  is 
the  Fonn  1067 — a  method  for  operational  and  maintenance  unit  commanders  to  work 
together  to  modify  a  specific  weapon  system  through  adding  bolt-on  equipment.  A  key 
Form  1067  limitation  is  that  the  solution  must  be  engineered  locally  effectively  reducing 
the  tradespace  of  hardware  solutions  to  non-core  functions  and  requiring  already 
established  interface  standards  for  software  solutions.  The  final  feedback  system  is  the 
annual  Weapons  and  Tactics  Conference  (WEPTAC).  While  these  conferences 
previously  captured  hardware  and  software  change  requests,  current  Air  Combat 
Command  WEPTACs  de-scoped  materiel  solution  discussion  and  focuses  instead  on 
Tactics  Improvement  Proposals  (TIP);  i.e.  how  to  optimally  use  what  we  currently  have. 
The  author  was  unable  to  discover  documentation  on  why  WEPTAC  de-scoped  material 
solutions. 
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The  Air  Force  has  one  feedback  system  that  can  accept  unsolicited  user  inputs: 
deficiency  reports  through  test  &  evaluation  squadrons.  Regretfully  operator  access  to 
and  knowledge  of  this  program  is  lacking.  This  program  documents  areas  where  the 
aircraft  does  not  meet  requirements;  however,  the  governing  directives  for  the  deficiency 
reporting  program  do  not  require  that  the  test  organizations  collect  deficiencies  from 
operators  nor  does  it  explicitly  define  a  deficiency.  Deficiencies  that  impact  safety  of 
flight  or  risk  mission  failure  are  defined;  however  small  problems,  especially  those  where 
the  specification  does  not  meet  the  needs  for  the  mission,  are  not  clearly  a  part  of  the 
undefined  tenn  “deficiency”  (AFI  99-103,  2013:86).  The  regulation  does  require  OT&E 
to  report  features  and  defects  of  recently  fielded  software  or  hardware.  One  such  briefing 
included  an  overview  of  future  projects,  a  list  of  open  deficiencies,  and  an  appeal  for 
operators  to  report  defects  in  the  new  software  since  one  full  day  of  combat  operations 
would  log  more  flight  hours  on  the  new  programming  than  all  OT&E  flights  combined. 

System  defects  are  a  subset  of  deficiencies;  however,  defects  make  up  the 
majority  of  known  deficiencies  in  the  MQ-1.  One  deficiency  report  highlights  the 
difference  between  a  deficiency  writ  large  and  a  defect.  Current  flight  software  has  a 
defect  where  the  distance  setting  for  an  automatic  trigger  only  works  as  specified  in  the 
two  most  common  control  modes  (local  control  and  remote-split  control)  but  not  the 
other  modes.  The  deficiency  report  accurately  describes  the  defect;  however,  it  misses  a 
deeper  problem:  the  trigger  distance  automatically  activates  both  overt  and  covert 
emitters.  The  combined  nature  of  the  programming  is  not  tactically  sound  over 
unfriendly  territory.  The  author  would  like  to  separate  a  trigger  for  each  of  the  emitters 
and  was  unaware  of  how  to  communicate  this  desire  until  perfonning  this  study. 
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Research  Obj  ectives/Questions/Goal 

The  end  goal  of  the  research  is  to  determine  if  the  Air  Force  should  establish  a 
formal  program  for  collecting  and  prioritizing  change  requests  from  operators,  and  if  so, 
how  should  the  process  be  implemented  and  what  characteristics  should  the  system 
possess.  Literature  review  will  highlight  ideal  characteristics  of  feedback  program  and 
then  evaluate  current  military  and  non-military  feedback  programs  against  those 
characteristics.  Next  the  research  will  use  a  Delphi  study  to  collect  expert  opinions  from 
the  operator  community,  synthesize  the  opinions  into  a  consensus,  and  finally,  determine 
the  need  for,  the  desired  characteristics  of,  and  basic  architecture  of,  an  operator-initiated 
feedback  system. 

Methodology 

The  research  will  be  a  Delphi  study  of  the  methods  to  capture,  assimilate, 
prioritize  and  approve  user-generated  change  requests.  A  Delphi  study  is  a  qualitative 
method  for  generating  a  consensus  of  opinion  from  a  group  of  experts  (Dalkey  and 
Helmer,  1963:459).  The  group  of  experts  is  subjected  to  iterative  rounds  of 
questionnaires  with  controlled  feedback  in  between  the  rounds.  Unlike  a  round-table 
discussion,  a  Delphi  study  keeps  the  identities  of  the  respondents  withheld  from  the  rest 
of  the  group.  The  latter  stipulation  seeks  to  harness  the  benefits  of  group  discussion  yet 
prevent  direct  confrontation  and  inevitable  bias  based  on  rank,  position,  passion,  or 
oratory  skills.  Subsequent  rounds  of  questions  aim  to  consolidate  the  opinion  and 
potentially  introduce  new  materiel  requested  by  an  expert  during  the  previous  round. 

A  key  component  of  the  Delphi  study  is  the  experts  selected  for  the  discussion. 
The  experts  for  this  discussion  will  be  limited  to  the  RPA  (Remotely  Piloted  Aircraft) 
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community  inside  of  Air  Combat  Command.  This  scope  allows  a  significant  but  not 
unwieldy  number  of  participants,  focuses  the  effort  on  a  new  and  expanding  enterprise, 
and  engages  a  command  solely  focused  on  equipping  mission  capability. 

The  Delphi  study  will  be  traditional  in  that  the  participants  will  be  anonymous 
and  not  directly  interact  with  each  other.  This  consideration  is  especially  important  as  the 
rank  and  position  difference  between  participants  is  projected  to  be  rather  high. 
Additionally,  the  study  has  the  potential  to  highlight  key  mindset  differences  between 
subcultures  inside  the  community.  Moderated  feedback  is  the  best  method  to  dissolve 
potential  friction  that  could  prevent  a  consolidated  consensus. 

Investigative  Questions 

As  a  Delphi  study,  the  initial  investigative  questions  will  be  directly  delivered  to  a 
group  of  experts.  The  exact  questionnaire  can  be  found  in  Appendix  B:  Initial 
Questionnaire;  however,  a  summary  of  the  essential  elements  is  discussed  here. 

This  study  seeks  to  determine  if  it  is  in  the  Air  Force’s  best  interest  to  establish  a 
program  to  constantly  accept  operator  inputs.  Specific  questions  will  determine  desired 
stakeholders,  the  proper  role  of  operators,  and  what  aspects  and  characteristics  of 
commercial  feedback  methods  are  desirable  in  the  Air  Force’s  RPA  community.  Other 
questions  seek  to  detennine  specific  implementation  details  such  as  the  proper 
communication  process,  decision  maker,  and  specific  details  captured  in  the  feedback. 

The  summation  of  the  questions  seeks  both  a  qualitative  and  quantitative  response  to  the 
potential  to  establish  a  new  program. 
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Assumptions/Limitations 

This  research  has  the  same  limitations  as  other  Delphi  studies.  Delphi  studies 
operate  under  the  assumption  that  multiple  experts  together  will  produce  a  better  result 
than  a  single  expert.  A  correlated  assumption  is  that  moderated  feedback  will  eventually 
produce  a  consensus  between  the  experts. 

The  major  limitation  is  the  scope  of  the  study:  a  specific  community  (MQ-1  and 
MQ-9)  further  narrowed  by  inclusion  of  only  one  major  command  inside  that  community 
(Air  Combat  Command).  The  study  and  the  concluding  recommendations  may  not 
perfectly  capture  the  needs  of  other  commands  or  agencies — those  that  train  and  equip  as 
well  as  those  that  execute  the  mission  such  as  Special  Operations  Command. 

Another  potential  limitation  is  the  unclassified  nature  of  the  report.  A  participant 
may  respond  based  on  an  experience  with  a  classified  program  and  the  unclassified 
response  would  lack  the  broader  context.  Additionally  some  potential  participants  may 
choose  to  not  participate  because  of  fear  of  discussing  any  mission-related  material  in  the 
unclassified  environment.  Cases  of  the  former  concern  might  be  mitigated  through 
additional  communication  at  higher  classification  levels;  however,  such  methods  increase 
the  risk  of  spillage  and  will  therefore  be  discouraged. 

Implications 

In  a  static  and  perfect  world,  requirements  would  never  change  and  the  current 
systems  engineering  process  would  fully  capture  the  needed  effects.  Neither  of  those 
stipulations  is  actually  correct.  First,  the  systems  engineering  process  is  not  perfect:  there 
will  be  some  errors  with  the  known  requirements.  A  recent  example  is  the  addition  of  an 
automated  weapons  engagement  zone  (WEZ)  display  for  the  MQ-1  Predator.  In  2013  a 
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new  software  revision  included  this  display;  however,  the  display  was  not  built  to  allow 
timely  updates  for  new  missile  variants  and  the  accurate  display  was  obsolete  prior  to 
operational  release  of  the  software.  Second,  the  operating  environment  is  not  static:  the 
nature  of  a  particular  mission  changes  over  time,  operators  are  constantly  discovering 
new  adversaries  and  environments,  and  strategic  planners  are  always  changing  the 
required  missions  for  airframes.  For  example,  the  Predator  aircraft’s  mission  started  with 
reconnaissance  in  the  Balkans,  then  added  Close  Air  Support  (CAS)  and  most  recently 
adding  Strike  Coordination  and  Reconnaissance  (SCAR)  missions  (“MQ-1B  Fact  Sheet,” 
2010;  Payette,  2005;  “Predator  IOC,”  2005).  Between  the  errors  in  capturing 
requirements  and  the  evolving  mission  sets,  the  known  requirements  of  a  weapon  system 
will  always  be  changing. 

This  study  will  increase  the  Air  Force’s  understanding  of  the  relationship  between 
end-users  and  requirements  generation.  A  key  difference  between  this  requirements 
study  and  most  others  is  the  origin  and  timing  of  the  requirements  generation.  Most 
studies  focus  on  communication  methods  initiated  by  acquisition  officers  or  engineers  to 
increase  the  accuracy,  thoroughness,  and  stability  of  known  requirements  during  initial 
development.  This  study  opens  discussion  on  formal,  continuous  feedback  methods  the 
end-user  could  initiate  to  update  or  correct  known  requirements. 

Preview 

The  following  chapters  detail  the  literature  background,  methodology,  analysis, 
and  results  of  the  study.  Chapter  2  reviews  academic  literature  on  feedback  methods, 
tools,  characteristics  and  case  studies;  current  military  feedback  systems  for  materiel  and 
non-materiel  feedback;  and  culminates  in  a  critique  of  reviewed  feedback  programs 


7 


against  ideal  characteristics.  Chapter  3  covers  the  methodology  used  to  detennine  the 
Delphi  panelists  and  surveys.  The  analysis  of  the  surveys  is  contained  Chapter  4, 
specifically  which  characteristics  and  implementation  constructs  the  panelists  were  able 
to  confirm;  reject;  or  neither  confirm  nor  reject.  The  final  conclusions,  including  specific 
recommendations,  are  presented  in  Chapter  5. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  describes  in  more  detail  a  literature  review  of  concepts  introduced 
earlier.  The  first  section  identifies  ideal  characteristics  of  a  feedback  system  based  upon 
relevant  literature  and  guiding  directives.  Next  is  a  review  of  how  the  DoD  generates 
requirements  and  a  review  of  DoD  fonnal  non-materiel  and  materiel  feedback  methods. 
Following  the  DoD  review  is  a  review  of  commercial  methodology  including  a  sales 
representative  interview,  three  feedback  tools  and  two  feedback  case  studies.  A  summary 
table  with  discussion  shows  which  characteristics  each  of  the  tools  meet  and  which  they 
do  not. 

Characteristics  of  a  Good  Feedback  Program 

Prior  to  determining  the  best  type  of  feedback  methodology  and  building  a 
program  to  execute  it,  the  characteristics  of  a  good  program  must  be  specified  and 
discussed.  The  following  categories  define  “good”  based  on  a  literature  review  covering 
feedback  concepts,  tools  and  case  studies  for  a  variety  of  end  users. 

A  feedback  program  should  exist 

The  most  difficult  part  of  designing  a  system  is  usually  the  proper  understanding 
of  the  requirements  to  meet  the  desired  effect  rather  than  the  design  of  a  system  that  can 
meet  the  proper  requirements  (Kujala,  Kauppinen,  Rekola,  2001 :49).  This  effect  is  most 
profound  during  initial  systems  engineering;  however,  as  the  environment  in  which  the 
system  operates  evolves,  user  feedback  ensures  the  system  is  constantly  improved  and 
adapted  to  the  current  environment  (Hansson,  Dittrich  and  Randall,  2006:5). 
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This  improvement  and  adaptation  requires  more  stakeholder  input  than  just  the 
developers  of  the  system.  System  designers  commonly  fall  in  the  trap  that  they  think 
they  know  everything  about  an  end  user  and  their  needs,  but  in  reality  they  do  not  know 
what  they  do  not  know  (Schneider,  2011 : 172).  In  an  academic  case  study,  experts 
identified  areas  they  anticipated  student  feedback — 44  in  all.  The  case  study  system 
allowed  any  student  to  add  extra  areas  if  the  students  desired  to  give  feedback  in  an  area 
not  identified  by  the  experts.  In  the  study,  43  of  44  of  the  expert-defined  areas  were 
effectively  used,  however  students  volunteered  an  additional  47  feedback  areas.  During 
the  feedback  review,  the  experts  realized  that  5  of  the  47  volunteered  feedback  areas  were 
worth  further  investigation  and  the  experts  had  initially  missed  these  feedback  areas. 

In  the  example  above,  the  hierarchy  between  the  experts  and  the  students  is  rather 
flat.  For  larger  bureaucracies,  there  may  be  several  layers  of  supervision  between  users 
and  the  requirements  experts.  In  bureaucracies,  this  characteristic  decomposes  into  two 
areas:  availability  and  advertisement  to  the  line  operator.  Example  programs  are 
described  later  in  this  chapter. 

Focuses  on  refinement  of  current  functionality 

Most  users  view  new  equipment  under  the  paradigm  of  the  current  system’s 
processes  and  assume  any  new  equipment  will  be  used  in  the  same  manner  as  the  old 
(Kujala,  Kauppinen,  Rekola,  2001 :49).  In  this  regard,  operators  are  keenly  aware  of 
current  processes  and  can  highlight  the  areas  with  the  highest  risk  of  mission  failure 
under  current  constructs.  Therefore,  the  feedback  program  should  discourage  discussion 
of  novel  concepts.  Novel  application  of  current  weapon  systems  falls  under  the  realm  of 
the  Tactics  Review  Board  (AFI  11-260,  201 1:7).  Novel  weapon  systems  designed  to 
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meet  new  operational  concerns  are  already  covered  with  current  JCIDS  processes  (JCIDS 
Manual,  2015:  C7-8). 

Clear  methodology  for  soliciting  and  processing  change  requests 

The  Software  Engineering  Body  Of  Knowledge  (SWEBOK)  captures  best 
practices  in  software  development.  The  latest  guide  recommends  four  basic  actions  for 
software  change  requests,  although  the  core  principles  of  the  steps  apply  equally  to  non¬ 
software  change  requests  (Champagne  and  April,  2014:pp  6-9).  The  SWEBOK 
recommended  steps  are:  originating  the  change  request  (requirement  update),  enforcing 
the  change  process  (review)  flow,  capturing  the  review  board’s  decision,  and  reporting 
change  process  infonnation.  Other  processes  have  clear  methods  for  change  requests  but 
the  SWEBOK  steps  demonstrate  an  example  of  the  basic  concepts  behind  clear 
methodology. 

Encourages  focusing  feedback  on  actionable  subjects 

Unconstrained  user  feedback  can  include  complaints  on  subjects  that  are  outside 
the  ability  for  the  recipients  to  change  or  outside  the  scope  of  the  system  at  hand 
(Schneider,  2011 : 165).  A  good  feedback  system  focuses  responses  towards  areas  that  are 
inside  the  available  tradespace  and  discourages  responses  that  are  not  (Schneider, 

2011 : 166).  This  focusing  could  be  as  simple  as  pre-defining  feedback  subject  areas, 
however  focusing  must  be  balanced  with  openness  to  unexpected,  but  potentially  valid 
feedback  subject  areas.  When  balanced  the  feedback  system  collects  useful  vice 
distracting  reviews. 
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Low  user  effort  required 

The  value  of  unsolicited  feedback  from  the  field  is  that  it  captures  the  fleeting 
moment  when  a  feedback  idea  is  conceived  rather  than  waiting  for  a  solicited  event.  To 
maximize  this  effect,  the  system  should  be  fast,  cheap  and  easy  for  the  user  Schneider, 
2011 : 166).  Barriers  to  the  fast,  cheap  and  easy  construct  only  serve  to  decrease  the 
available  feedback  to  the  engineers.  One  method  to  decrease  the  effort  is  to  pre-define 
feedback  subject  areas  and  to  provide  common  or  “canned”  feedback  messages.  The  case 
study  used  drop-down  lists  of  entities  available  for  feedback  such  as  “Lecture  on  software 
modeling,”  “Usability  Room,”  or  “Online  Registration  System”;  pre-defined  modes  such 
as  “complaint,”  “complement,”  or  “neutral  comment”;  and  pre-defined  options  such  as 
“not  very  usable”  or  “confusing  presentation”  (Schneider,  2011 : 170). 

Automated  triage  of  feedback 

The  nature  of  soliciting  feedback  continuously  invites  an  opportunity  for  an 
overwhelming  number  of  feedback  entries  (Gartner  and  Schneider,  2012:47).  In  order  to 
effectively  process  the  feedback  without  an  undue  requirement  for  a  human’s  time,  an 
ideal  system  would  automate  some  of  the  initial  triage  of  inputs.  A  prototyped  technique 
for  accomplishing  the  triage  recommends  counting  both  keywords  and  critical  keywords 
then  analyzing  the  frequency  of  the  keywords  in  a  specific  feedback  message  compared 
to  all  feedback  messages.  This  technique  accommodates  multimedia  attachments; 
however,  this  feature  was  not  tested  in  the  case  study. 

Similar  medium  between  feedback  and  object  of  the  feedback 

In  a  parallel  to  the  adage  “a  picture  is  worth  a  thousand  words”  feedback  that  is 
not  in  the  same  medium  as  the  object  of  the  feedback  is  prone  to  unnecessary 
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communication  errors  (Rashid,  2007:372).  The  referenced  article  studied  the 
applicability  of  using  graphical  feedback  methods  to  capture  change  requests  for  a 
computer’s  graphical  interface.  Similarly,  any  feedback  system  must  encourage  users  to 
submit  feedback  in  the  proper  medium;  text,  chart,  table,  diagram,  screen  shot,  and  video 
are  all  potential  means  of  communication.  Feedback  systems  that  accept  various 
mediums  of  capturing  the  requirement  decrease  the  amount  of  effort  required  from  the 
user  and  promote  more  user  involvement  (Schneider,  2011 : 166). 

Accommodating  various  mediums  of  user  feedback  is  especially  important  due  to 
trends  in  feedback.  Generally  speaking,  users  tend  to  seek  improvements  to  the  interface 
than  the  structure  or  functionality  of  the  system  (Kujala,  Kauppinen,  Rekola,  46).  The 
value  of  improving  the  interface  is  displayed  in  the  F-22  cockpit:  the  entire  cockpit  was 
designed  to  present  the  proper  type  and  amount  of  information  to  the  pilot  in  an  easy  to 
comprehend  method  (“F-22  cockpit,”  2015).  Allowing  users  to  submit  interface 
feedback  in  a  graphical,  pictorial,  or  videographic  could  assist  in  improving  the  interface 
quality. 

Captures  user  context,  environment,  situation  or  background 

A  key  objective  of  requirements  elicitation  is  to  understand  the  user’s  perspective. 
One  of  the  first  steps  to  understanding  the  user  is  to  establish  a  user  profile  and  capture 
the  domain  of  the  system  under  development  or  change  (Perez  and  Valderas,  2009:5).  As 
an  adaptation  of  participatory  design,  feedback  systems  should  segregate  users  based  on 
common  interests;  in  practice  this  means  establishing  multiple  user  profiles  (Hansson, 
Dittrich  and  Randall,  2006: 179).  Each  profile  should  include  infonnation  covering  the 
general  skill,  mindset  or  culture  of  the  individuals  using  the  system  (Perez  and  Valderas, 


13 


2009:5).  Finally,  a  specific  user  profile  should  automatically  be  associated  the  individual 
submitting  the  feedback  without  burdening  the  user  to  constantly  generate  their  own 
profile  (Schneider,  2011:166). 

Domain  infonnation  captures  the  physical,  information,  or  social  environment  the 
system  intends  to  operate  (Perez  and  Valderas,  2009:33).  The  domain  context  also 
should  include  any  adaptation  in  system  behavior  when  subjected  to  different  domains, 
such  as  the  difference  between  the  engagement  area,  the  en  route  transit  between 
engagement  area  and  the  airfield,  and  the  terminal  airfield  (Knauss,  2012:346).  Like  the 
user  context,  this  information  is  best  when  captured  automatically;  e.g.  the  system 
captures  the  time,  date  and  physical  location  of  the  user  when  submitting  feedback 
(Schneider  2011 : 166).  Capturing  the  context  assists  designers  developing  more  complete 
and  accurate  requirements. 

Captures  complete,  consistent  requirements 

Users  typically  submit  feedback  in  the  fonn  of  natural  language  that  allows  for 
incomplete,  ambiguous  or  internally  inconsistent  requirements  (Pinto-Albuquerque  and 
Rashid:  2014,  233).  A  review  for  incomplete  requirements  should  check  the  chain  of 
events  from  input  to  desired  output;  this  process  can  be  partially  automated  if  the  desired 
function  is  modeled  in  Unified  Modeling  Language  and  evaluated  using  an  Event-drive 
Process  Chain  (Knauss,  Lubke,  Meyer,  2009:589).  A  second  review  of  completeness 
should  ensure  the  initiator  captured  any  changes  to  the  following  elements  of  a 
requirement:  constraints,  user  activities,  data  flow,  quality  and  role  of  the  requirement 
(John  and  Dorr,  2003:5).  Additionally,  six  specific  heuristics  can  check  requirements  for 
ambiguity  and  inconsistency  (Pinto-Albuquerque  and  Rashid,  2014:236-238).  Ambiguity 
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heuristics  look  for  imprecise  words,  alternate  grammatical  constructs,  and  alternate 
contexts.  Inconsistency  heuristics  search  for  quantified  requirements  that  do  not  match 
related  or  dependent  requirements.  The  precise  heuristics  used  to  check  for  consistency 
is  not  as  important  as  the  presence  and  effectiveness  of  a  consistency  review  process. 

Highlights  potential  for  unintended  consequences 

Change  requests  based  on  one  scenario  may  impact  the  way  the  system  responds 
in  other  scenarios.  The  same  inconsistency  heuristics  intended  to  check  for  consistency 
inside  of  a  new  requirement  request  can  also  highlight  all  the  areas  any  proposed  change 
would  change  system  characteristics  (Pinto-Albuquerque  and  Rashid,  2014:233).  This 
allows  any  decision  maker  to  properly  assess  the  change  prior  to  approval  or 
prioritization. 

Feedback  initiators  must  trust  their  inputs  have  impact 

Users  providing  feedback  have  an  intrinsic  desire  to  share  their  experiences  and  to 
improve  the  system  they  use  (Schneider,  2011 : 166).  In  order  to  use  product 
improvement  as  a  motivator,  the  user  must  trust  that  their  inputs  have  an  impact  on  the 
final  product.  The  principle  of  impact  is  shared  with  participatory  design,  however 
participatory  design  expands  the  definition  to  ensure  that  feedback  participants  are  not 
hanned  due  to  the  participation  (Hansson,  Dittrich,  and  Randall,  2006:179).  Ultimately, 
this  impact  is  shown  through  changes  to  the  final  product;  however,  the  initiator  should 
get  a  response  from  the  decision  authority  on  the  final  status  of  the  feedback — approved, 
partially  approved,  or  rejected. 
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Current  Military  Requirements  Methodology 

The  military  has  several  methods  for  determining  an  individual  system’s 
requirements.  The  Joint  Capabilities  Integration  and  Development  System  initially 
identifies  the  requirements.  During  the  lifetime  of  the  system  the  Joint  Lessons  Learned 
Program,  AF1067,  and  Deficiency  Reports  refine  the  initial  needs. 

Joint  Capabilities  Integration  and  Development  System  (JCIDS) 

The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  governs  the 
process  of  generating  requirements  for  high  level  (DoD  or  service  specific)  military 
systems  (CJCSI  3 170.011,  2015: 1).  At  this  macro  level,  the  initial  step  is  detennining  a 
capability  gap.  These  capability  gaps  are  identified  in  a  variety  of  methods:  Capabilities- 
Based  Assessments;  development  of  Operation  Plans  (OPLANS)  and  Concept  Plans 
(CONPLANS)  including  Joint  Urgent  Operational  Needs  (JUONs)  and  Urgent 
Operational  Needs  (UONs);  exercise  or  warfighting  lessons  learned;  and  technology 
demonstrations  (JCIDS  Manual,  2015:C3-8).  Once  the  capability  gap  is  discovered,  the 
organization  discovering  the  gap  must  formally  review  the  gap  to  detennine  the 
appropriate  response.  The  JCIDS  Manual  guides  the  decision  based  on  validating  the  gap 
and  assessing  available  assets  as  depicted  in  Figure  1.  During  the  annual  Capability  Gap 
Assessment  (CGA),  these  capability  gaps  are  reviewed  and  stratified  at  the  DoD-level 
(JCIDS  Manual,  2015:BA1). 
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Figure  1.  Decision  Tree  For  Capability  Gaps  (JCIDS  Manual,  2015:C9) 


Both  the  above  process  and  the  stratification  are  strategic  in  nature  and  cover  the 
whole  range  of  potential  responses  to  a  capability  gap.  What  is  notably  downplayed  is 
the  role  of  the  line  operator.  Of  the  handful  of  methods  to  discover  a  capability  gap,  only 
the  ‘lessons  learned’  has  potential  for  tactical  operators  to  directly  contribute. 

Joint  Lessons  Learned  Program  (JLLP) 

Unfortunately,  the  current  construct  of  the  Joint  Lessons  Learned  Program  (JLLP) 
does  not  lend  itself  to  dissemination  at  the  lowest  levels.  The  JLLP’s  stated  mission 
implies  operator  involvement  through,  among  other  activities,  discovery  and 
dissemination  of  lessons  across  a  wide  variety  of  joint  operations  (CJCSI  3150.25, 
2012:Al-2).  This  mission  is  delegated  down  to  the  service  level  (AFI  90-1601,  2013:5). 
While  the  program  “encourages”  all  airmen  to  participate,  the  program  is  postured  to 
have  direct  involvement  only  from  the  Lesson  Learned  team  with  indirect  involvement 
from  operators  (AFI  90-1601,  2013: 1 1).  A  brief  survey  of  lesson  learned  content  in  the 
Joint  Lessons  Learned  Infonnation  System  (JLLIS)  shows  a  recording  of  a  variety  of 
Doctrine,  Organization,  Training,  materiel,  Leadership,  Policy  and  Education,  Personnel, 
Facilities,  and  Policy  (DOTmLPF-P)  information,  however  the  information  is  oriented 
toward  operational-level  infonnation  not  system-level  infonnation.  In  short,  the  JCIDS 
process,  even  with  the  Joint  Lesson  Learned  Program,  does  not  capture  operator-level 
information  in  a  fonn  that  communicates  the  specific  feedback  that  operators  have,  but 
are  unable  to  deliver. 
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Air  Force  Form  1067  (AF1067) 

Detailed  operator  input  is  needed  for  the  proper  development  of  a  System 
Requirements  Document  (SRD).  The  purpose  of  the  SRD  is  to  translate  an  operational 
capability  gap  into  acquisition  requirements  and  is  the  responsibility  of  the  DoD,  not  the 
contractor  (Mil-HDBK-520,  2010:4;  DAG,  2013:1 143).  In  a  parallel  of  the  ways  to 
discover  a  capability  gap,  there  is  only  one  tactical-level  method  to  make  inputs  into  the 
SRD  development  process:  the  Air  Force  Form  1067,  Modification  Proposal  (AFI  10- 
601,  2013:8).  While  the  SRD  process  has  accurately  captured  that  operators  request  new 
capabilities  through  the  AFI 067,  this  input  method  is  hardly  optimal  as  the  AF1067’s 
primary  purpose  is  not  to  capture  a  capability  gap  (AFI  63-131,  2013:21).  The 
mismatched  purposes  between  the  AFI 067  and  the  SRD  make  the  acquisition  officer’s 
job  of  incorporating  operator  input  into  the  SRD  challenging. 

The  Modification  Proposal  form,  AFI 067,  is  intended  to  request  permission  to 
modify  a  configuration  item  such  as  a  weapon  system  as  stated  on  the  actual  form 
duplicated  in  Appendix  E:  Air  Force  Fonn  1067.  As  a  tool  to  capture  a  capability  gap, 
the  process  is  categorically  flawed:  AF  1067  only  captures  capability  gaps  to  which  an 
operator  has  already  found  a  likely  solution  and  which  funding  has  already  been 
eannarked  (AFI  63-131,  2013:22).  In  this  regard,  the  AFI 067  is  an  effective  tool  to 
ensure  the  functionality  of  a  weapon  system  is  not  impacted  by  a  unit-requested 
modification  of  the  system,  but  other  capability  gaps  are  not  addressed. 

AFI 067  has  been  used  with  mixed  results  in  the  MQ-1  community.  A  successful 
example  is  the  mounting  of  new  monitor  brackets  for  supplemental  computer  screens  in 
the  MQ-1  Ground  Control  Station  (GCS).  The  original  bracket  configuration  used 
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identical  placement  of  computer  screen  mounts  between  several  versions  of  GCSs.  One 
particular  model  of  GCS  benefitted  from  moving  the  brackets  to  a  new  location  to 
provide  the  operators  more  physical  space  (Goldsmith,  2011).  The  AF1067  allowed  the 
squadron  to  formally  request  and  Air  Combat  Command  (ACC)  and  the  MQ-1  System 
Program  Office  (SPO)  to  formally  approve  the  bracket  move. 


Figure  2.  Picture  of  Ground  Control  Station  Brackets 

A  later  AF1067  was  less  successful.  After  a  trend  of  near  misses  between  MQ-1 
and  other  aircraft  in  theater,  the  20th  Reconnaissance  Squadron  Commander  looked  for 
methods  to  alert  pilots  of  factor  air  traffic  (Goldsmith,  2011).  Using  a  datalink  tool  on 
supplemental  computers,  aircrew  had  visual  representation  of  the  air  picture.  The 


20 


supplemental  computer  could  generate  an  audio  signal  as  an  alert,  however  the  computer 
was  not  integrated  into  the  GCS  audio  panel.  Using  organic  funding,  the  operators  and 
maintainers  built  a  simple,  modest-capability  modification  to  the  GCS  communication 
panel.  The  MQ-1  SPO  approved  a  temporary  AF1067  for  testing  the  new  modification. 
Although  the  test  showed  the  modification  performed  as  expected,  the  permanent 
AF1067  was  rejected  since  a  new  acquisition  program  promised  a  timely  development  of 
an  advanced  audio  suite  with  expanded  and  significantly  more  complex  requirements. 
Unfortunately  for  the  line  operator,  the  advanced  suite  experienced  programmatic  delays 
from  a  late  discovery  of  a  non-functional  security  requirement  while  the  simple  solution 
was  never  approved  nor  implemented. 

Cockpit  Working  Group 

The  Cockpit  Working  Group  (CWG)  is  a  group  of  various  stakeholders,  including 
operators,  that  advise  an  MDS’s  Program  Manager  with  both  technical  guidance  and  an 
operational  perspective  (AFI  63-1 12,  2011 :2).  While  the  Lead  Command  selects  senior 
aircrew  to  represent  two  staff  agencies,  line  operators  represent  the  MDS’s  current  or 
future  operator  community,  specifically  as  Cockpit  Evaluation  Team  members  (AFI  63- 
1 12,  2011 :3).  The  CWG  meets  regularly  and  makes  recommendations  for  cockpit 
modifications  to  the  Program  Manager. 

Deficiency  Reports 

The  last  example  of  materiel  feedback  systems  in  the  military  is  the  deficiency 
report.  Unfortunately  “deficiency”  is  not  explicitly  defined  in  either  the  governing 
technical  order  or  the  parent  instruction.  Various  sub-sets  of  deficiencies  are  defined  and 
their  summation  says  a  deficiency  is  a  quality,  materiel,  software,  warranty,  or 


21 


informational  condition  that  is  unsafe  or  limits  the  use  of  the  materiel  for  purpose 
intended  (TO  00-35D-54,  201  l:pp  1-5  to  1-10).  The  deficiencies  are  discovered  during 
inspections,  engineering  reviews  or  dedicated  Test  &  Evaluations  (TO  00-35D-54, 

2011 : 1-4  to  1-5).  The  deficiency’s  originator  submits  the  deficiency  to  the  OT&E  or 
DT&E  Test  Director  who  performs  the  initial  review  and  prioritization  (TO  00-35D-54, 
2011 :2-l).  The  Test  Director  submits  the  deficiency  to  the  Program  Manager.  The 
deficiencies  are  reviewed  and  prioritized  at  a  Review  Board  including  the  Test  Director, 
Program  Manager,  a  Lead  MAJCOM  representative  and  vested  organizations  inside  the 
testing  agency  (TO  00-35D-54,  2011 :2-2  to  2-3).  The  Program  Manager,  a  member  of 
the  System  Program  Office  and  therefore  Material  Command,  chairs  the  board  (TO  00- 
35D-54,  2011 :4- 12).  As  an  example  the  RPA  Test  Director  and  the  MAJCOM 
representative  both  report  to  the  Lead  MAJCOM  (“53rd  Wing,”  2015).  The  decisions 
from  the  review  board  are  updated  in  a  dedicated  database  (TO  00-35D-54,  201 1:4-13). 

The  above  process  is  oriented  towards  Test  &  Evaluation  of  operational 
conditions  rather  than  end  users  in  the  actual  field.  Officially  a  deficiency’s  originator  is 
“any  individual”  who  discovers  a  deficiency  and  program  managers  must  process 
Deficiency  Reports  (DR)  “originating  from  any  source,”  which  could  include  end  users 
operating  aircraft  in  the  field;  however,  the  publication  only  requires  action  from  testing 
organizations  (TO  00-35D54,  201 1:1-9  and  AFI  99-103,  2013:59).  Additionally,  the 
regulations  focus  on  the  process  to  meet  user  needs  but  have  a  hierarchal  definition  of 
user: 

“User:  Refers  to  the  operating  command  which  is  the  primary  command 
operating  a  system,  subsystem,  or  item  of  equipment.  Generally  applies  to 
those  operational  commands  or  organizations  designated  by  Headquarters, 
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US  Air  Force  to  conduct  or  participate  in  operations  or  operational  testing, 
interchangeable  with  the  term  "using  command"  or  “operator.”  In  other 
forums  the  term  “warfighter”  or  “customer”  is  often  used.  (AFI  10-601) 

Also  refers  to  maintainers.”  (AFI  99-103,  2013:98) 

The  different  definition  implies  that  the  end  users  are  able  to  use  their  chain-of-command 

to  affect  materiel  change  or  that,  as  described  in  a  later  survey  response,  OT&E  is  the 

“voice  of  the  operator”  to  both  engineers  and  the  MAJCOM  leadership. 

Informal  Methods 

The  previous  discussion  focused  on  the  formal  methods  of  determining 
requirements — what  is  not  covered  is  informal  methods  of  feedback.  By  nature,  these 
feedback  methods  are  not  structured  and  often  only  include  the  operator  delivering 
feedback  to  someone  who  can  influence  the  system  design.  Responses  from  the  infonnal 
process  are  not  always  transmitted  back  to  the  operator  involved.  One  final  example 
from  the  20RS  highlights  the  informal  methods.  An  operator  noticed  that  the  baseline 
GCSs  did  not  have  a  particular  software  program  present  on  a  prototype-tumed- 
operational  version  of  the  GCS  (Goldsmith,  2011).  The  operator  casually  mentioned  to 
the  lead  maintainer — a  civilian  working  for  the  supplier — that  the  program  would  aid 
operator  awareness  of  the  aircraft  and  that  the  program  was  already  built;  it  would  just 
need  to  be  installed  on  the  baseline  models  as  well.  Approximately  four  months  later,  the 
program  was  installed  as  part  of  a  previously  scheduled  software  update.  The  operator 
does  not  know  if  the  installation  was  a  result  of  the  conversation,  which  highlights  the 
problem  with  the  informal  nature.  Had  the  program  not  been  installed,  the  operator 
would  never  have  known  why  the  request  was  rejected. 
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The  previous  review  shows  the  formal  methods  found  in  documentation  as  well  as 
a  case  of  a  potential  informal  method.  The  focus  so  far  has  been  solely  on  military 
requirements  generation  leading  to  materiel  solution.  The  next  section  covers  programs 
in  the  military  that  accept  operator  feedback,  however  they  do  not  give  inputs  to  the 
requirements  process. 

Non-Materiel  Military  Feedback  Methods 

The  Air  Force  currently  runs  other  feedback  methods  for  non-materiel  solutions. 
There  are  two  feedback  methods  that  focus  on  the  documentation  and  procedures  of 
operating  or  maintaining  an  aircraft  or  a  weapon  system  and  two  feedback  methods  that 
focus  on  tactics.  The  feedback  methods  for  the  publications  are  the  Air  Force  Technical 
Order  Form  22  (AFT022)  and  the  Air  Force  Fonn  847  (AF847).  Through  these  two 
forms,  stakeholders  can  recommend  changes  to  guidance  that  require  approval  up  the 
chain  of  command  ending  at  either  a  System  Program  Office  or  a  staff  agency  potentially 
as  high  as  the  Headquarters  of  the  Air  Force  (HAF).  The  feedback  methods  for  tactics 
updates  are  both  conferences:  the  Tactics  Improvement  Proposal  and  the  Weapons  and 
Tactics  Conference.  These  programs  are  detailed  below. 

Air  Force  Form  84  7 

The  AF847  is  the  official  Air  Force  fonn  for  changing  a  publication  (Appendix  D: 
Air  Force  Fonn  847).  The  header  information  of  the  fonn  shows  the  breath  of  coverage: 
feedback  on  any  Air  Force  Instruction  (AFI)  or  aircrew  Technical  Order  (TO)  is 
acceptable.  A  Technical  Order  is  the  authoritative  source  of  infonnation  and  procedures 
for  operating  or  maintaining  an  aircraft.  The  fonn  can  have  many  originators  but  line 
operators  and  investigation  boards  as  the  most  common  source  for  change  requests  to 
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both  TOs  and  operation-oriented  AFIs  (Program  Office,  2014).  The  content  of  the  form 
allows  the  originator  to  detail  both  the  desired  change  and  the  rationale  behind  the 
change.  Additionally,  the  originator  lists  contact  infonnation  and  forwards  the  change 
request  up  their  chain  of  command  (AFI  1  l-202v2,  2012: 12).  To  be  more  precise, 
commanders  in  the  chain  delegate  Form  847  supervision  to  their  unit  Standardization  and 
Evaluation  (Stan-Eval)  shops  and  the  originator  forwards  the  request  first  to  the  unit 
Stan-Eval  shop.  For  TOs,  the  request  moves  up  the  Stan-Eval  chain  to  the  MAJCOM 
level  then  over  to  the  Flight  Manual  Manager  at  the  System  Program  Office  (SPO)  for 
technical  review  (AFI  1 1-215,  201 1 :40).  For  publications,  the  request  is  up-channeled 
until  reaching  the  publication’s  Office  of  Primary  Responsibility  (OPR)  as  defined  for 
each  publication.  This  OPR  could  be  as  high  as  the  HAF  staff,  such  as  AFI  1  l-202v3, 
General  Flight  Rules,  whose  OPR  is  the  Headquarters  Air  Force  Flight  Standards  Agency 
(AFI  1  l-202v3,  2014:1).  For  either  type  of  change  recommendation,  the  originator’s 
chain  can  reject  the  recommendation  at  any  level  and  the  originators  are  notified  if  their 
requests  are  rejected.  Once  at  either  the  SPO  or  the  OPR,  recommendations  may  be 
deferred  while  awaiting  additional  analysis  or  closed  with  acceptance  or  rejection.  Once 
closed,  the  OPR  or  SPO  contacts  the  originator  with  the  final  detennination,  thus 
finishing  the  communication  loop.  Although  limited  to  documentation  and  procedures, 
the  AF847  program  shows  one  method  for  capturing,  processing  and  communicating 
change  requests. 

Air  Force  Technical  Order  Form  22 

AFT022  is  a  very  similar  form  to  the  AF847,  however  its  purpose  is  to  update 
maintenance  TOs  rather  than  operator  TOs  (Appendix  F:  Air  Force  Technical  Order 
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Form  22)(TO  00-5-1,  2014:1-1).  The  AFT022  program  follows  the  same  basic  flow:  the 
originator  is  typically  a  line  maintainer  who  submits  the  change  recommendation  to  the 
chain  of  command  who  reviews  the  recommendation  until  an  evaluator  makes  a 
detennination.  Again,  upon  closure  the  originator  receives  feedback  on  the  final  status. 
Despite  the  similarities,  AFT022  includes  a  significant  difference:  a  discussion  on  the 
predicted  savings  in  tenns  of  both  dollars  and  man-hours. 

This  addition  speaks  to  the  nature  of  maintenance  vice  operations:  maintenance 
change  requests  are  directly  linked  to  the  bottom  line.  Maintenance  units  track  the 
money  and  manpower  required  to  sustain  a  concrete  metric  for  aircraft  ability  rate. 
Incremental  improvements  to  the  process  will  accrete  into  real  savings  through  either 
materiel  or  manpower  reductions.  The  same  is  not  true  for  operations:  incremental 
improvements  may  expand  the  upper  bound  of  a  system’s  effectiveness,  but  the  upper 
bound  is  not  often  needed.  An  operator  effectiveness  metric  is  difficult  to  quantify  and 
incremental  improvements  rarely  results  in  the  elimination  of  a  whole  crew  position  and 
the  associated  manning  requirement. 

Although  maintenance  change  requests  are  more  directly  involved  with  saving 
money  or  time,  neither  process  allows  for  changes  to  the  aircraft  to  improve  its 
operational  characteristics  or  its  maintainability.  Both  the  maintainer  and  the  operator  are 
largely  unable  to  initiate  formal  feedback  on  the  weapon  system  under  their  care.  This 
circumstance  is  mirrored  in  the  civilian  sector  with  notable  exceptions  in  the  field  of 
software. 
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Weapons  and  Tactics  Conference 

The  first  of  two  Weapons  &  Tactics  feedback  methods  is  the  Air  Reserve 
Component  (ARC)  Weapons  and  Tactics  Conference  (WEPTAC).  The  ARC  WEPTAC 
is  a  method  for  the  ARC  to  prioritize  capability  gaps  that  require  materiel  or  tactical 
solutions  to  properly  steward  limited  financial  resources  (Vest,  2015).  A  select  few 
members  of  each  line  squadron,  commonly  the  squadron  commander  and  the  weapons 
officer,  typically  attend  ARC  WEPTACs.  At  the  conference,  these  selected  stakeholders 
represent  their  squadrons  in  prioritizing  both  materiel  solutions  and  tactical  testing  with 
the  intent  to  defeat  the  highest  threat. 

In  contrast,  current  Air  Combat  Command  WEPTACs  do  not  focus  on  materiel 
solutions.  This  is  a  change  from  past  WEPTACs  that  aimed  to  capture  some  feedback  on 
materiel  shortfalls  (Goldsmith,  2010).  ACC  WEPTACs  are  focused  on  changing  tactics 
to  meet  operational  capability  gaps;  however  informal  discussions  of  materiel  solutions 
may  occur.  The  informal  discussions  may  lead  to  either  the  MDS  SPO  or  requirements 
Action  Officer  creating  a  Air  Systems  Requirements  Council  (ASRC)  (Vest,  2015).  An 
ASRC  seeks  operator  input  through  voting  on  specific  modification  options  such  as 
airframe  upgrades  or  pilot  mechanization.  The  SPO  or  the  Action  Officers  may  request 
wing  involvement  via  representatives,  commonly  a  weapons  officer. 

Tactics  Improvement  Proposal 

The  last  military  method  of  feedback  is  the  Tactics  Improvement  Proposal  (TIP). 
A  TIP  is  a  non-materiel  potential  solution  to  a  tactical  deficiency  (AFI  1 1-260,  201 1:3). 
Users  submit  the  TIP  to  the  unit-level  Weapons  &  Tactics  shop.  A  TIP  includes  a 
description  of  the  tactical  problem,  a  recommended  solution,  and  a  recommended  testing 
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plan  with  objectives.  The  review  chain  is  through  squadron,  group/wing,  Numbered  Air 
Force,  MAJCOM  then  finally  to  the  Combat  Air  Forces.  The  CAF  Tactics  Review 
Board  (TRB)  consists  of  eight  to  ten  people  from  MAJCOM  staff,  Test  &  Evaluation  and 
operator  communities  (AFI  1 1-260,  201 1:7).  Any  MAJCOM,  including  PACAF, 
USAFE,  and  AETC,  may  request  representation  at  the  TRB  from  the  chair.  The 
regulation  does  not  require  general  announcement  of  the  TRB  results  but  the  results  of 
the  ACC  TRB  are  broadcast  to  the  ACC  WEPTAC — an  audience  that  typically  has 
representation  from  each  unit  and  at  every  level  from  squadron  to  MAJCOM  leadership. 
Current  Non-Military  Methodology 

The  literature  review  detailed  the  role  of  sales  representatives,  two  case  studies  of 
specific  agencies  that  collected  and  processed  change  requests,  three  specific  tools  for 
gathering  change  requests,  and  multiple  methods  to  condense  broad  data  into  actionable 
information.  In  general,  the  available  articles  discussing  user-initiated  feedback  for 
software  products  vastly  overwhelmed  documentation  of  user-initiated  feedback  for 
physical  products.  Fortunately  the  lopsided  representation  does  not  impact  the  utility  of 
the  literature  review:  the  concepts  detailed  in  the  articles  apply  to  both  software  and 
hardware  development. 

A  summary  of  the  research  is  presented  below,  starting  with  the  sales 
representative  interview,  then  the  case  studies  and  finally  covering  available  tools.  The 
sales  representative  interview  presents  a  method  for  physical  product  feedback.  The  first 
case  study  documents  the  change  process  for  the  Space  Shuttle  flight  software — a 
government  program  with  low  tolerance  for  failure  (DiVito  and  Roberts,  1996:3).  The 
case  study  is  actually  focused  on  Space  Shuttle  integration  with  GPS,  but  the  whole 
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change  request  process  is  summarized  to  provide  context  for  the  GPS  modifications.  The 
other  case  study  details  Idavall  Data  AB,  a  six-employee  European  business  serving  1300 
users  from  300  organizations,  mostly  municipal  governments  (Hansson,  Dittrich,  and 
Randall,  2006: 1).  The  three  tools  are  also  significantly  different.  ConTexter  is  a 
smartphone  application  that  records  semi-structured  messages  that  could  capture 
feedback  on  any  program,  organization  or  product  (Schneider,  2011 : 166).  OpenProposal 
is  a  graphical-oriented  tool  intended  on  capturing  Graphical  User  Interface  (GUI)  change 
requests  (Rashid,  2007:372).  Finally,  techniques  to  data-mine  vast  numbers  of  online 
customer  reviews  are  reviewed  (Somprasertsri  and  Lalitrojwong,  2010:938;  and  Zhang, 
Narayanan,  and  Choudhary,  2010:1). 

Sales  Representatives 

Many  manufacturers  of  physical  products  use  sales  representatives.  A  sales 
representative  typically  works  in  an  independent  firm  and  represents  one  or  more 
companies  on  various  lines  of  products.  An  interview  with  the  owner  of  a  sales 
representative  firm  for  heat  exchangers  and  other  chemical  process  equipment  revealed 
representatives  also  personify  the  customer  when  talking  to  the  manufacturers 
(Bourgeois,  2015).  The  sales  representative  informally  collects  equipment  feedback 
through  on-site  visits  after  initial  installation  and  maintains  communication  through 
telephone  or  email.  The  customers  rarely  communicate  independently  to  the 
manufacturers  except  for  marketing  and  sales  related  surveys.  When  customers  do  talk 
directly  to  engineers,  the  sales  representative  is  present — the  sales  representatives 
normally  discourage  exceptions  to  this  cultural  nonn.  In  this  manner,  sales 
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representatives  are  often  the  voice  of  the  manufacturer  to  the  customer  and  the  voice  of 
the  customer  to  the  manufacturer. 

NASA  flight  software  changes 

The  NASA  study  showed  that  individual  engineers  initiate  change  requests  with 
direct  communication  to  other  stakeholders,  starting  with  written  communication  to  a 
software  requirement  analyst  (DeVito  and  Roberts,  1996:9).  The  software  requirements 
analyst  performs  an  informal  review  and  returns  the  change  request  with  comments.  The 
engineer  may  iteratively  ask  for  multiple  infonnal  reviews,  however  once  the  engineer 
feels  the  request  is  correct,  he  or  she  submits  the  change  request  directly  to  a  formal 
review  board. 

The  review  board  then  prioritizes  the  fonnal  submissions  to  undergo  more 
scrutiny  (DeVito  and  Roberts,  1996:9).  The  fonnal  inspector  follows  a  checklist  of  past 
problems  to  avoid  and  has  periodic  meetings  with  the  stakeholders  to  ensure  consistent 
understanding  of  the  change  request.  Issues  revealed  during  review  are  considered  open 
until  determined  to  not  be  a  problem  or  until  a  solution  is  found. 

The  NASA  case  study  documented  a  few  problems  with  the  review  system  as 
stated  above.  First,  the  inspectors  had  little  methodology  to  perfonn  the  review — the  list 
of  past  errors  was  not  structured  enough  to  guide  reviews  (DeVito  and  Roberts,  1996:10). 
Second,  the  review  had  no  completion  criteria — thoroughness  was  open  to  individual 
variance.  Third,  there  is  no  structured  method  to  document  the  depth  of  review,  the 
understanding  required  to  process  the  review,  or  the  good  aspects  of  the  request.  These 
three  review  deficiencies  serve  as  a  great  lesson  learned  for  future  projects. 
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Once  the  review  board  certifies  all  the  open  issues  with  a  change  request  are 
resolved,  the  change  proposal  is  implemented;  i.e.  it  is  coded  and  incorporated  into  the 
baseline  software.  No  formal  communication  to  the  other  stakeholders  is  documented, 
although  informal  methods,  such  as  supervisor  feedback  to  the  initiator,  may  occur 
without  being  documented  (DeVito  and  Roberts,  1996:10).  The  change  request  is  now 
closed. 

Idavall  Software  Firm 

The  second  case  study  has  different  methods  for  accepting  user  inputs.  The 
Idavall  staff  uses  multiple  direct  methods  to  capture  needs  from  their  users:  helpdesk 
support  calls,  user  meetings,  and  instructional  courses  (Hansson,  Dittrich,  and  Randall, 
2006: 177).  Users  generally  call  the  helpdesk  when  problems  in  the  current  software  arise 
and  the  discussion  often  elicits  a  need  for  a  new  function.  Additionally,  all  employees 
field  helpdesk  phone  line,  including  developers,  and  hear  user  problems  and  mindset 
first-hand.  Idavall  also  hosts  eight-to-ten  user  meetings  annually  across  three  countries  to 
informally  disseminate  news,  discuss  future  development,  answer  questions,  and 
generally  establish  a  user  community.  The  Idavall  hosts  encourage  users  to  present  new 
proposals  at  the  meetings.  Finally,  Idavall  conducts  user  classes,  as  the  program  requires 
some  formal  training.  Like  the  user  meetings,  the  teachers  encourage  students  to  submit 
feedback.  In  short,  Idavall  uses  direct  communication  to  capture  change  requests  from  a 
wide  field  of  users  mostly  via  phone  call  or  in-person  meetings.  Extended  iterative 
discussions  were  not  present  in  the  case  study,  but  Idavall  staff  constantly  interacted  with 
their  users. 
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The  review  process  for  Idavall  is  significantly  more  informal — planned  meetings 
are  rare,  however  infonnal  meetings  of  the  six  employees  occur  often  over  coffee  and 
lunch  breaks  (Hansson,  Dittrich,  and  Randall,  2006:178).  At  these  meetings  change 
requests  are  ranked  according  to  the  universality,  secondary  effects,  longevity,  and 
impact  of  the  change.  The  meeting  concludes  with  an  implementation  determination. 
Like  the  NASA  case  study,  no  formal  method  of  disseminating  results  was  listed. 
Although  the  company  disseminates  news  via  its  website,  newsletter,  and  user  meetings 
but  no  formal  communication  of  implementation  decisions,  especially  rejected  requests, 
are  documented  (Hansson,  Dittrich,  and  Randall,  2006:175). 

Contexter  Feedback  Tool 

The  first  feedback  tool,  ConTexter,  also  relies  on  direct  communication,  however 
the  developers  seek  to  focus  the  stream  of  feedback  to  into  usable  distinctions  based  on 
the  context  of  the  feedback  (Schneider,  2011 : 168).  Specifically  the  tool  allows 
developers  to  pre-define  entities  that  may  receive  feedback  but  also  allows  users  to 
specify  new  entities.  The  entities  are  either  physical  items — such  as  computers,  rooms, 
or  weapon  systems — or  abstract  elements — like  lectures,  organizations  or  job 
designations.  After  specifying  the  entity,  the  user  then  selects  the  type  of  comment 
(complaint,  compliment,  mixed  or  neutral).  Finally  the  application  allows  the  user  to 
freely  type  the  comment.  Additionally  the  application  records  context  such  as  the  last 
website  accessed  and  the  physical  location  of  the  device.  When  completed,  the 
application  sends  a  message  to  the  entity’s  owner,  if  a  pre-defined  entity  is  selected,  or  to 
the  entire  review  board  if  the  entity  has  not  been  defined.  The  scope  of  this  tool  ends 
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once  the  entity  owner  or  the  review  board  receives  the  message;  it  does  not  assist  with 
review  of  the  comment  nor  communicates  the  final  detennination  to  the  originator. 

OpenProposal  Feedback  Tool 

Another  tool,  OpenProposal,  focuses  on  allowing  graphical,  not  textual,  user 
feedback  on  the  system’s  GUI  followed  by  a  period  of  collaboration  on  the  change 
request  (Rashid,  2007:372).  The  user  feedback  portion  allows  the  user  to  take  a  screen 
shot  of  the  current  system  and  to  annotate  specific  requirements.  The  user’s  annotations 
are  treated  as  individual  objects  with  amplifying  details  on  the  exact  problem  and  the 
desired  solution  (Asarnusch,  Wiesenberger,  Meder,  and  Baumann,  2009:16).  Once 
annotated,  the  proposal  is  saved  in  a  database  and  is  available  to  all  stakeholders,  thus 
OpenProposal  is  another  method  of  direct  communication,  but  the  core  message  is 
graphical  with  annotated  text. 

The  OpenProposal  tool  goes  beyond  collection  of  the  requirement  and  also 
facilitates  collaborative  discussion  between  all  stakeholders,  including  users,  requirement 
analysts  and  software  engineers  (Asarnusch,  Wiesenberger,  Meder,  and  Baumann, 

2009: 15).  Once  submitted,  the  change  request  is  stored  in  a  database  based  primarily  on 
which  software  object  the  change  seeks  to  modify.  Stakeholders  access  the  database  via 
the  submission  program  (summary  list  only)  and  via  a  specially  designed  webpage.  This 
website  uses  filters  for  specific  users,  historical  web-addresses,  and  active  applications  to 
prevent  information  overload.  After  selecting  a  particular  request,  stakeholders  can 
discuss  the  change  request  with  each  other.  Specifically,  end  users  can  clarify 
requirements,  requirement  analysts  can  infer  desirability,  and  software  engineers  can 
detennine  feasibility.  OpenProposal  does  not  currently  offer  any  formal  review  tools, 
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just  the  stakeholder  discussion,  to  assist  the  requirements  analyst  in  determining  which 
changes  to  implement  and  which  to  reject.  Additionally,  the  tool  did  not  discuss  any 
formal  communication  contracts  between  the  stakeholders,  just  that  the  tool  offers  a 
method  to  have  the  communication. 

Data  Mining 

The  final  set  of  tools  identified  in  the  literature  for  processing  user  feedback  is 
various  different  techniques  for  data  mining  online  reviews  of  products  (Somprasertsri 
and  Lalitrojwong,  2010:938;  and  Zhang,  Narayanan,  and  Choudhary,  2010:1).  These 
tools  allow  producers  to  take  advantage  of  the  feedback  online  retailers  are  already 
collecting  to  assist  fellow  customers.  These  techniques  revolve  around  semantic 
dissection  then  aggregation  of  all  customer  comments.  The  strength  of  data  mining  is  the 
ability  to  reduce  large  quantities  of  raw  data  into  applicable  summaries;  in  fact  the  tools 
are  only  suited  for  that  application.  Data  mining  does  not  seek  to  capture,  discuss,  or 
transmit  change  decision;  it  just  is  a  method  to  analyze  current  feedback. 

Critiques  of  Current  Feedback  Methods 

A  summary  of  evaluating  the  reviewed  feedback  mechanisms  against  the 
characteristics  of  a  good  program  previously  reported  is  seen  below  as  Table  1.  The  table 
graphically  shows  the  strengths  and  limitations  of  the  reviewed  feedback  methods. 
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Table  1:  Summary  of  Current  Feedback  Critiques 


Available  to  /  designed  for  any  line  operator 

Advertised  to  all  line  operators 

Focuses  on  refining  current  functionality 

Captures  complete  &  consistent  requirements 

Highlights  unintended  consequences 

Clear  methodology 

Automated  triage  of  feedback 

Accommodates  various  feedback  mediums 

Captures  user  context/environment 

Low  user  effort  required 

Focuses  feedback  on  actionable  subjects 

Initiators  trust  inputs  have  impact 

AF  1067 

X 

P‘ 

X 

P2 

P3 

P4 

X 

AF  847 

X 

X 

X 

P2 

P3 

P4 

X 

AFTO  22 

X 

X 

X 

P2 

P3 

P4 

X 

JLLIS 

X 

X 

P3 

X 

Deficiency  Reports 

X 

P9 

X 

X 

X 

P2 

P3 

p7 

Cockpit  Working  Grp 

9 

p 

X 

X 

X 

X 

X 

ARC  WEPTAC 

p9 

X 

X 

X 

X 

TIP 

p9 

X 

X 

X 

X 

X 

Sales  Representative 

X 

X 

X 

p8 

X 

X 

p8 

NASA  Space  Shuttle 

X 

X 

X 

X 

X 

X 

Idavall 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Contexter 

X 

* 

X 

X 

X 

X 

X 

Open  Proposal 

X 

* 

X 

X 

P6 

X 

X 

X 

p5 

X 

Data  mining 

* 

X 

X 

X 

X=fully  meets  the  characteristic 
p=partially  meets  the  characteristic 
*=not  applicable 

p1  -  this  method  has  significant  limitations  on  changing  system  functionality 
p2  -  these  methods  allow  non-video  attachments 

p3  -  these  methods  have  free  text  entry  areas  that  may  include  context  if  the  initiator  is  aware  of 
the  importance  of  context 

p4  -  these  methods  have  readily  available  assistance  from  program  managers 

p5  -  this  program  elicits  more  detailed  and  precise  feedback  than  other  programs;  however,  the 

program  intends  to  ease  the  process  as  much  as  possible 

p6  -  this  program  currently  only  supports  informal  reviews  of  the  requirement 

p7-  initiators  are  typically  have  database  access  as  they  are  not  line  operators 

p8  -  most  feedback  is  informal  and  relies  on  individual  skill-level 

p9  -  initiator  commonly  has  informal  contact  with  majority  of  unit-level  operators 
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This  summary  reveals  several  key  observations.  First,  no  current  method  has  all 
the  characteristics,  although  the  Idavall  program  and  OpenProposal  meet  more 
characteristics  than  the  others.  Idavall’s  strength  primary  comes  from  the  direct,  iterative 
interaction  between  developers  and  users;  this  attribute  accounts  for  six  of  their  seven 
strengths.  OpenProposal’s  strength  is  similar,  however  the  direct,  iterative  interaction  is 
accomplished  digitally.  OpenProposal’s  interaction  includes  three  important 
stakeholders:  requirement  analysts,  software  engineers,  and  users  interact  with  each 
other.  The  requirement  analyst  serves  to  address  non-engineering  limitations  and  sheds 
light  on  the  process  for  getting  the  user’s  concerns  addressed. 

Secondly,  the  summary  shows  that  many  programs  meet  a  characteristic  only  for 
a  specific  scope  or  under  specific  circumstances.  This  is  most  prevalent  with  the  military 
feedback  fonns — each  form  is  manually  entered  and  has  multiple  free-text  entry  areas. 
The  manual  entry  means  the  user  has  to  spend  some  effort  finding  the  proper  data  for  the 
manual  entry  vice  easier  automation.  Additionally,  the  free-text  areas  allow  great 
flexibility  and  great  potential  for  less-than-complete  data  entry.  These  partial  areas  could 
be  made  into  full  areas  with  some  modifications. 

Summary  and  Way  Ahead 

The  above  literature  review  describes  current  feedback  systems  and  concepts  in 
the  DoD  and  the  commercial  sector.  The  chapter  culminates  with  a  summary  table 
comparing  current  feedback  methodologies  with  identified  ideal  characteristics.  These 
characteristics  were  based  on  case  studies  and  other  literature.  The  comparison  evaluated 
six  DoD  feedback  programs  and  six  non-DoD  tools  and  methods. 
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The  remainder  of  the  research  will  examine  which  of  the  listed  characteristics  the 


operational  RPA  community  consider  important  plus  detennine  basic  architecture  of  a 
desirable  feedback  system.  The  literature  review  broadened  the  field  of  characteristics 
included  in  the  examination  and  provided  a  variety  of  examples  of  architectures.  After 
examination,  the  literature  provides  examples  for  how  to  implement  or  further  study 
characteristics  or  architectures  deemed  operationally  important  but  lacking  in  current 
military  feedback  programs. 
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III.  Methodology 


Chapter  Overview 

The  selection  and  execution  of  the  research  methodology  is  dependent  on  the 
research  goals  and  limitations.  The  goal  of  the  research  is  to  detennine  the  operator- 
desired  characteristics  and  relative  importance  of  operator-generated  feedback  without 
actually  implementing  the  feedback  program.  This  study  employed  the  Delphi  study 
method:  a  qualitative  study  method  best  suited  for  subject  area  with  a  lack  of  historical 
data  and  an  inability  to  run  experimental  tests.  The  remainder  of  the  chapter  covers  the 
justification  for  the  Delphi  study  method,  an  expanded  description  of  the  Delphi  method 
and  the  execution  details — panelist  selection,  open-ended  survey  development,  follow-on 
survey  development  seeking  specific  answers,  consensus  definition,  and  the  termination 
decision  for  panelist  involvement  via  surveys. 

Why  Delphi  Method? 

The  purpose  of  this  study  is  to  detennine  the  desired  characteristics  and  relative 
importance  of  operator-generated  feedback  on  Air  Force  weapon  systems.  This  study 
does  not  seek  to  implement  a  specific  feedback  system  and  therefore  is  unable  to 
experimentally  derive  conclusive  data  on  the  utility  of  such  a  system.  Additionally,  the 
lack  of  historical  data  prevents  traditional  statistical  analysis.  The  lack  of  experimental  or 
historical  data  is  a  key  condition  for  implementing  a  Delphi  Study  (Rowe  and  Wright, 
2001 : 135).  Fortunately  the  Delphi  Method  is  exceptionally  useful  in  situations  lacking 
conclusive  data;  specifically  it  evokes  sharing  and  processing  the  collective  knowledge 
and  experiences  of  the  expert  panelists  (Powell,  2003:380). 
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Delphi  Study  Method  Overview 

The  Delphi  method  relies  on  the  proper  selection  of  experts  composing  the  panel. 
The  first  criterion  for  an  expert  is  the  willingness  and  ability  to  make  a  useful 
contribution  to  the  discussion  (Powell,  2003:379).  In  the  military,  willingness  to 
participate  in  studies  is  rarely  a  problem;  being  useful  has  different  challenges.  The  two 
key  factors  in  planning  useful  participants  are  to  ensure  individual  members  have  the 
proper  domain  knowledge  and  the  collective  knowledge  and  expertise  spans  the  full 
scope  of  the  research  (Rowe  and  Wright,  2001 : 127).  Additionally,  the  research  is  more 
accurate  when  heterogeneous  members  are  combined:  this  applies  to  both  the  different 
perspectives  of  the  problem  set  and  varying  personalities  of  the  members  (Powell, 
2003:379). 

Expert  Panel  Selection 

The  expert  panel  is  focused  on  the  RPA  community,  specifically  defined  as  the 
MQ-1  Predator  and  MQ-9  Reaper.  Five  perspectives  relate  to  the  requirements 
generation  process  and  its  impact:  the  line  operator,  the  weapon  school,  the  test  & 
evaluation  squadron,  the  system  program  office,  and  ACC’s  Directorate  of  Plans, 
Programs  and  Requirements  (ACC  A5/8/9).  Despite  the  general  lack  of  requirements 
generation  process  knowledge  inside  the  line  operator  community,  a  panel  discussing  line 
operator  inputs  should  include  their  perspective  to  fully  cover  the  scope  of  the  topic.  To 
mitigate  the  knowledge  gap,  line  operators  with  previous  experience  with  another 
perspective  were  selected.  The  26th  Weapons  School  (26WPS)  is  the  pinnacle  of  tactical 
expertise  and  the  hub  of  emerging  combat  capability  for  the  RPA  community. 
Additionally  they  host  the  RPA  working  group  for  the  ACC  WEPTACs.  The  556th  TES 
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provides  developmental  testing  to  the  RPA  community  and  briefs  the  community  on 
aircraft  or  flight  software  modifications — they  are  most  visual  segment  of  the 
acquisitions  process  to  line  operators.  The  MQ-1  and  MQ-9  SPOs  are  the  link  to  the 
engineers  who  would  be  responsible  for  producing  and  manufacturing  any  modifications 
to  the  aircraft.  Finally,  the  ACC  A5/8/9  is  the  lead  major  command  (MAJCOM)  for  RPA 
and,  as  such,  the  final  authority  on  RPA  requirements.  These  five  perspectives 
encapsulate  the  requirements  generation  process  and  its  relationship  with  line  operators. 

There  are  two  additional  considerations  for  selection.  Fortuitously,  one  of  the 
operators  also  assisted  the  Air  Force  Scientific  Advisory  Board  (SAB)  as  an  Executive 
Officer.  The  SAB  advises  the  Secretary  of  the  Air  Force  and  the  Chief  of  Staff  of  the  Air 
Force  through  identifying  technology  that  can  improve  or  create  Air  Force  capabilities 
(USAF  Scientific  Advisory  Board:  2015).  The  other  consideration  is  the  desire  to  have  a 
broad  array  of  ranks,  crew  positions,  and  operational  backgrounds.  RPA  have  two 
crewmembers:  a  pilot  and  a  sensor  operator.  Sensor  operators  are  effectively,  but  not 
officially,  enlisted  aviators.  Their  collective  expertise  includes  the  operator,  test  & 
evaluation,  and  some  SPO  interaction.  Sensor  operators  with  appropriate  knowledge  are 
typically  Technical  Sergeants  (TSgt)  or  higher.  Pilots  are  rated  officers  who  span  all  of 
the  perspectives  and  generally  meet  a  high  level  of  knowledge  as  a  senior  Captain. 
Experienced  RPA  pilots  generally  have  completed  either  pilot  training  or  navigator 
training  prior  to  cross-training  as  an  RPA  pilot.  New  RPA  pilots  only  have  experience 
with  RPA;  however,  this  community  is  not  represented  due  to  the  relative  inexperience  in 
the  airframe.  Finally,  the  line  operators  were  selected  from  the  RPA  schoolhouse  at 
Holloman.  This  scope  is  deliberate:  schoolhouse  members  are  combat  experienced;  have 
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exposure  to  multiple  operational  theaters,  varied  types  of  acquisition  support,  and  the 
doctrinal  mission  requirements;  are  more  open  to  share  in  an  unclassified  environment; 
and  have  a  reduced  operational  tempo  leading  to  a  higher  likelihood  of  survey 
completion. 

Once  selected  each  panelist  was  assigned  a  phonetic  alphabet  codename  in  order 
discuss  specific  responses  while  protecting  the  panelist’s  actual  identity.  For  example, 
“Panelist  Charlie”  has  advised  the  Scientific  Advisory  Board.  The  order  of  the  panelists 
does  not  follow  any  particular  convention.  This  naming  convention  is  consistent 
throughout  the  survey  analysis  and  conclusion. 

Goal  of  the  Questionnaires 

The  first  questionnaire  is  aimed  at  qualitatively  identifying  the  specific  topics  of 
discussion  for  the  later  rounds  (Delphi  Technique  Myths  and  Realities,  3).  This  initial 
response  will  validate  feedback  system  characteristics  as  described  in  Chapter  2.  Further 
refinement  of  the  panel’s  responses  in  subsequent  rounds  of  questioning  will  transition 
from  qualitative  to  quantitative  assessment  of  the  topics,  mainly  through  generating  a 
prioritized  listing  of  key  stakeholders,  roles,  and  attributes.  In  addition  to  the  lists, 
subsequent  rounds  seek  to  identify  a  potential  feedback  methodology  for  the  RPA 
community  and  verify  the  key  characteristics  as  delineated  in  Table  1  at  the  end  of 
Chapter  2.  The  goal  is  to  define  both  the  key  aspects  and  identify  a  potential  way  to 
execute  a  feedback  program  based  on  those  aspects. 

First  Questionnaire 

The  questions  inside  the  first  panel  survey  are  deliberately  open-ended  and  allow 
the  participants  to  answer  freely  on  the  topic  of  the  survey  (Powell,  2003:378).  These 
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types  of  questions  are  designed  to  elicit  responses  that  can  be  qualitatively  analyzed;  the 
analysis  identifies  specific  discussion  points  for  future  rounds.  The  previous  literature 
review  has  a  limited  role  in  the  first  round:  concepts  discovered  bound  the  topic’s 
discussion  to  a  manageable  and  meaningful  scope. 

The  initial  survey  (see  Appendix  B:  Initial  Questionnaire)  seeks  to  capture  the 
intersection  between  line  operator  inputs  and  how  the  Air  Force  generates  requirements. 
Initially  the  survey  captures  key  demographic  infonnation:  education  level,  job  title, 
flying  qualifications,  flying  experience  (measured  in  hours  of  military  flight  time),  years 
flying  in  an  operational  squadron,  and  any  interaction  with  the  acquisition  system.  After 
the  demographics,  the  survey  examines  the  topic  at  hand.  The  five  content  questions  are 
listed  below: 

1 .  Imagine  that  an  Air  Force  system  or  Major  Design  Series  (MDS)  has  recently 
reached  IOC  and  is  now  used  operationally.  What  stakeholders  (individual  job 
positions  or  communities  of  people)  should  have  power  to  change  the  system  to 
be  more  effective  in  future  operations  and  why?  Please  list  in  priority  order  and 
include  any  discussion  or  justification  you  feel  necessary. 

2.  What  is  your  personal  view  on  the  proper  role  of  end-users  or  line  operators  in  the 
system  modification  or  upgrade  process? 

3.  If  you  could  change  one  thing  about  the  methods  or  process  used  to  determine  a 
system’s  requirements  inside  the  DoD  what  would  it  be? 

4.  What  are  your  top  2  or  3  preferred  characteristics  of  any  feedback  system? 

5.  What  type  of  information  should  any  feedback  system  seek  to  capture? 

The  first  question  is  from  the  acquisition  community’s  perspective:  as  they 

manage  a  system’s  capabilities,  how  should  they  weigh  the  inputs  from  various 
stakeholders.  The  second  question  is  from  a  user’s  perspective:  how  should  they  interact 
with  the  other  stakeholders.  The  third  question  looks  at  the  entire  requirements 


42 


generation  system  and  seeks  to  identify  improvement  areas.  Like  the  first  question, 
answers  to  Question  3  seek  to  show  the  relative  importance  of  user  input  compared  to 
other  competing  improvement  efforts.  The  last  two  questions  cover  the  ideal  feedback 
system  capturing  both  the  non-functional  requirements  (Question  4)  and  the  functional 
requirements  (Question  5). 

Converting  qualitative  responses  into  quantitative  questions  relies  on  statistical 
methods.  The  first  round  of  questions  has  three  distinct  types  of  responses  each  requiring 
different  analysis:  prioritized,  variable-length  lists;  non-prioritized  lists;  and  unstructured, 
verbose  prose.  The  non-prioritized  lists  are  the  easiest  to  quantify:  the  number  of  times  a 
particular  response  is  mentioned  is  summed  across  all  responses.  Similar  responses  are 
combined;  i.e.  similar  means  the  responses  had  common  or  synonymous  keywords. 

Prose  responses  will  first  be  decomposed  into  individual  phrases.  These  phrases  will  then 
be  treated  the  same  as  the  non-prioritized  lists.  The  detennination  of  similar  phrases  may 
be  a  more  significant  challenge  than  with  a  list;  however,  phrases  with  similar  verbs  and 
adverbs  will  be  merged  into  one  new  phrase  capturing  the  essence  of  both  contributory 
phrases.  Quantifying  a  set  of  prioritized  and  variable-length  lists  is  more  of  a  challenge 
than  the  other  situations.  To  properly  steward  the  participant’s  time,  the  panel  will  rank- 
order,  but  not  individually  weigh  each  item.  Each  item  will  then  be  assigned  a  weight-of- 
importance  percentage  according  to  the  following  formula: 

Weightn  =  (1) 

Zjc=i  c 

Where  N  is  the  size  of  the  participant’s  list  and  n  is  the  rank  of  the  item  in  the  list 

For  example,  a  three-item  list  will  have  50%  weight  on  the  first  item,  33%  on  the 

second  item  and  17%  on  the  third  item.  Once  all  the  individual  lists  are  weighted,  similar 
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items  across  the  entire  panel’s  lists  will  receive  a  summed  weight,  by  which  the  final, 
combined  list  is  sorted. 

Subsequent  Surveys 

Subsequent  surveys  transition  from  open  ended  questions  to  specific  questions  to 
continue  validation  of  the  key  characteristics  in  Table  1  and  identify  a  useful  feedback 
methodology  for  the  RPA  community.  Questions  1-4  of  Survey  2  roughly  align  with  the 
four  basic  steps  of  SWEBOK  software  change  methodology  described  earlier  in  Chapter 
2  linked  here:  Clear  methodology  for  soliciting  and  processing  change  requests. 

Question  5  of  Survey  2  seeks  to  capture  the  important  elements  for  complete  and 
thorough  feedback.  The  questions  will  also  avoid  areas  that  already  have  consensus  to 
focus  on  concepts  that  need  more  exploration  to  reach  consensus.  The  entire  panel’s 
open-ended  responses  to  Survey  1  will  become  the  possible  selections  for  questions  in 
Survey  2.  Survey  2’s  main  questions  are  listed  below  with  the  entire  survey  found  in 
Appendix  C:  Second  and  Final  Questionnaire. 

1.  User  involvement:  Virtually  all  surveys  indicated  users  should  be  involved  in 
change  requests  to  some  degree.  What  user  types  of  user  involvement  should  be 
accepted? 

2.  Final  detennination:  (a)  Who  should  be  the  final  approval  authority  for  change 
requests?  Assume  the  commander  in  question  may  delegate  authority  to  a  lower 
staff  member  for  minor  change  requests,  (b)  Many  responses  indicated  that 
decisions  should  not  be  made  in  a  vacuum  and  other  stakeholders  should  be  able 
to  influence  the  final  decision  on  change  requests.  Who  of  the  following  should 
have  influence  or  give  suggestions  about  the  change  request?  Check  all  that  apply. 

3.  Vetting  Process:  Several  responses  indicated  the  need  to  filter  feedback.  The  most 
common  Air  Force  method  is  to  have  a  functional  chain  of  command  sequentially 
review  submissions.  For  example,  the  Form  847  is  sequentially  processed  from 
the  user  to  Squadron  Standardization/Evaluation  to  Group  Stan/Eval  to  NAF 
Stan/Eval  to  MAJCOM  Stan/Eval.  In  contrast,  a  prominent  commercial  feedback 
model  uses  a  group  discussion  between  the  user,  requirements  analysts,  and 
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engineers  to  fully  capture  the  requirement.  Which  of  the  following  proposed 
methods  would  BEST  facilitate  timely  and  complete  reviews  of  submissions? 

4.  Non-functional  characteristics:  About  half  of  the  responses  indicated  the  feedback 
program  should  have  good  communication.  Which  of  the  following  common 
architectures  of  data  repositories  would  BEST  assist  good  communication  without 
sacrificing  timeliness  or  inducing  waste  by  excess  communication? 

5.  Specific  feedback  items:  If  a  feedback  program  were  to  exist,  it  should  effectively 
communicate  the  change  request  and  the  complete  context  around  the  request. 
Rank  order  the  following  potential  items  based  on  their  ability  to  support 
communication. 

Along  with  the  quantitative  analysis,  panelists  are  encouraged  to  provide 
justification  for  their  responses  in  at  least  one  question.  Subsequent  surveys  will  include 
both  the  quantitative  analysis  and  the  collective  considerations  from  the  panel  (expert 
opinions  in  forecasting,  3).  The  inclusion  of  the  rationales  seeks  to  increase  response 
accuracy  through  informing  all  panelists  of  potential  aspects  that  might  not  have  been 
considered  while  answering  the  first  survey. 

Additionally,  the  second  questionnaire  used  the  “Form”  feature  of  Adobe  Pro. 
Once  configured,  Adobe  Pro  can  import  individual  surveys,  collate  the  responses,  and 
export  the  results  as  a  single  file.  This  method  also  reduces  the  effort  required  from 
survey  recipients  due  to  pre-fonnatted  spaces  for  responses  and  the  ability  to  digitally 
sign  the  initial  release. 

Role  of  the  User 

In  Question  5  of  Survey  1 ,  panelists  described  the  role  of  the  end  user.  The  vast 
majority  described  some  method  for  end  users  to  provide  feedback;  however,  the  exact 
method  varied  significantly.  Question  1  of  Survey  2  combined  the  answers  from  Survey 
1  and  presented  them  as  possible  selections  to  determine  which  types  of  user  interaction 
are  desired.  Panelists  could  select  more  than  one  option  if  desired. 
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Decision  maker  and  influencing  members 

Question  4  of  Survey  1  provided  a  wide  array  of  potential  stakeholders  in  the 
change  process.  Question  2  of  Survey  2  had  two  parts  to  capture  the  different  aspects  of 
stakeholders:  the  final  decision  authority  and  acceptable  influence  to  that  decision 
authority.  In  Part  A,  panelists  had  a  choice  between  a  two  separate  sole  decision  makers 
or  a  council.  Members  who  selected  “council  vote”  chose  two-to-four  voting  members 
from  a  choice  of  four  commanders.  In  Part  B,  panelists  selected  as  many  other 
stakeholders  identified  in  Survey  1  as  they  felt  appropriate  to  influence  the  final  decision 
maker. 

Vetting  Process 

Several  responses  to  Survey  1  indicated  the  need  to  filter  line  operator’s  feedback. 
Question  3  of  Survey  2  polled  the  panel  for  the  best  feedback  method  out  of  three 
sequential  flows  and  two  group  discussion  flows.  The  panel  could  also  write-in  their  own 
flow.  The  basis  for  the  options  originated  in  Chapter  2  with  specific  inputs  from  Survey 
1 .  A  sequential  vetting  process  occurs  in  most  of  the  DoD  feedback  methods  described  in 
Chapter  2.  The  group  discussion  vetting  process  is  detailed  as  part  of  the  OpenProposal 
feedback  tool. 

Desired  Characteristics 

Questions  6  and  7  of  Survey  1  allowed  the  panel  to  give  open-ended  feedback  on 
current  feedback  methods.  Question  6 — the  #1  item  to  change  about  requirements 
generation — focused  on  the  negative  aspects.  Question  7 — desired  characteristics  of  a 
feedback  program — focused  on  the  positive  aspects.  Together  the  largest  area  of 
uncertainty  is  the  best  way  to  accommodate  both  timeliness  and  good  communication. 
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Under  the  assumption  that  change  requests  would  be  tracked  via  database  should  a 
feedback  program  exist,  Question  4  of  Survey  2  polled  the  panel  on  the  best 
implementation  of  a  timely  and  communicative  database. 

Specific  Feedback  Items 

The  last  question  in  both  Surveys  identified  then  prioritized  all  the  individual 
items  of  feedback  that  should  be  included  in  a  change  request.  The  responses  to  Survey  1 
were  condensed  into  thirteen  specific  areas  of  feedback  that  conceptually  cover  the  entire 
range  of  responses  to  Survey  1 .  Panelists  ordered  the  group  according  to  how  a  specific 
item  would  assist  communication  from  the  operator  filling  out  the  form  to  the  decision 
maker  deciding  on  the  overall  input. 

Terminating  the  Survey  Rounds 

The  main  intent  of  subsequent  surveys  is  to  reach  a  consensus  among  the 
participants  (Ryynanen,  Karvoneni,  and  Kassi,  2008: 1477).  Consensus  for  this  study  will 
be  defined  as  the  responses  show  stability  rather  than  attempting  to  reach  a  set  percentage 
of  agreement  between  the  panelists  (Rowe  and  Wright,  2001 : 128).  This  research  is 
largely  semantic  rather  than  numeric,  so  variability  cannot  be  measured.  Instead,  stability 
will  be  qualitative  differences  between  survey  responses.  This  definition  sets  a  realistic 
goal,  curbs  the  desire  to  add  extra  rounds  to  force  agreements,  and  serves  to  eliminate 
additional  pressure  for  participants  to  confonn  to  the  group  averages. 

In  addition  to  overall  consensus,  the  response  distribution  should  be  checked  for 
bi-modal  characteristics,  especially  given  the  distinct  groups  present  in  the  panel  (Powell, 
2003:379).  Bi-modal  responses  show  a  lack  of  consensus  that  needs  further 
investigation,  especially  if  the  peaks  correlate  to  distinct  participant  groups.  This  analysis 
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may  be  difficult  due  to  the  decreased  size  of  individual  panelist  but  could  reveal  biases 
inside  the  sub-groups. 

Summary 

The  previous  sections  detail  the  Delphi  method  and  how  it  will  be  implemented 
for  this  research.  The  end  goal  is  to  capture  the  answer  to  the  investigative  question 
presented  in  Chapter  1 :  what  are  desired  characteristics  and  relative  importance  of 
operator-generated  feedback  in  the  RPA  community.  The  answers  to  the  overarching 
question  will  come  from  an  expert  panel  spanning  the  entire  requirements  generation 
perspectives  for  RPA.  Specifically,  surveys  will  seek  consensus  among  the  entire  panel 
and  will  cease  when  consensus  is  either  made  or  assessed  unlikely  to  be  reached.  Once 
the  surveys  are  tenninated,  data  collection  is  complete. 
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IV.  Analysis  and  Results 


Chapter  Overview 

The  surveys  were  designed  to  capture  the  collective  judgment  of  the  panel  on 
whether  the  Air  Force  should  establish  a  program  to  collect  unsolicited  operator 
feedback,  and  if  so,  how  should  the  program  operate.  The  first  section  reviews  the  actual 
demographics  and  response  metrics  of  the  panel.  Next  the  results  of  the  surveys  are 
broken  into  three  major  areas:  consensus  areas,  non-consensus  areas,  and  correlation 
between  the  panelist’s  desired  attributes  and  the  ideal  characteristics  identified  in  Chapter 
2.  The  panel  further  explored  two  specific  characteristics — clear  methodology  and 
capturing  complete  requirements. 

Panel  Demographics  and  Responsiveness 

The  panel  started  with  fourteen  members  who  indicated  they  were  willing  and 
able  to  respond  to  survey  requests.  Thirteen  panelists  completed  Survey  1  and  ten 
panelists  completed  Survey  2.  The  demographics  are  shown  below.  Many  demographics 
show  a  numerical  bias  towards  officer-pilots  with  operational  experience  in  conventional 
operations,  this  bias  is  due  to  the  majority  of  members  in  special  assignments,  such  as 
TES,  having  the  above-mentioned  common  background. 
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Table  2:  Panel  Demographics — Acquisitions  Perspective 


Acquisitions  Perspective 
(panelists  may  have  more 
than  one  perspective) 

Accepted  Panel 
Invitation 

Responded  to 
Survey  1 

Responded  to 
Survey  2 

RPA  Operator 

11 

11 

9 

Weapon  School 

2 

2 

1 

Staff 

3 

3 

3 

System  Program  Office 

3 

2 

0 

Test  &  Eval  Squadron 

4 

4 

4 

Table  2  shows  the  coverage  across  the  five  different  acquisitions  perspectives  that 
span  the  requirements  generation  process.  The  large  amount  of  operators  was 
unavoidable:  the  majority  of  members  in  other  perspectives  had  previous  experience  as  a 
line  operator.  While  none  of  the  panel  members  with  SPO  experience,  including  an 
operator  with  SPO  liaison  experience,  responded  to  Survey  2,  significant  portions  of  their 
perspective  were  communicated  with  two  responses  from  Survey  1 .  However,  the  lack  of 
SPO  representation  in  the  final  survey  is  an  area  worth  further  investigation. 


Table  3:  Panel  Demographics — Currently  Assigned  Unit 


Currently  Assigned 

Accepted  Panel 
Invitation 

Responded  to 
Survey  1 

Responded  to 
Survey  2 

6 

6 

4 

3 

3 

3 

3 

3 

3 

2 

1 

0 

Table  3  closely  follows  the  previous  table.  While  qualitative  in  nature  due  to  the 
small  sample  size,  the  similarities  indicate  there  may  be  only  a  small  amount  of  cross¬ 
pollination  between  the  perspectives.  Many  panelists  shared  the  operator  perspective 
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with  another  perspective  but  no  panelists  shared  multiple  perspectives  between  WPS, 
TES,  SPO  or  staff.  Additionally  coverage  of  the  various  operational  units  required  the 
high  amount  of  operational  unit  members. 


Table  4:  Panel  Demographics — Operational  Experience 


Operational  Background 

Accepted  Panel 
Invitation 

Responded  to 
Survey  1 

Responded  to 
Survey  2 

Conventional  Operations 

8 

8 

8 

Special  Operations 

2 

2 

1 

Unconventional 

Operations 

2 

2 

1 

Unknown 

1 

1 

1 

Table  4  shows  the  different  types  of  operational  experience  present  in  the  panel. 
The  significance  of  the  operational  background  is  linked  directly  to  different  sources  of 
funding  and  acquisitions  for  equipment.  The  high  presence  of  conventional  operations  is 
mainly  due  to  the  desire  for  operators  to  have  other  perspectives  and  the  high  correlation 
between  TES  and  Staff  operators  with  only  conventional  experience.  Two  members 
shared  operational  experience  between  conventional  operations  and  other  backgrounds. 


Table  5:  Panel  Demographics — Rank 


Rank 

Accepted  Panel 
Invitation 

Responded  to 
Survey  1 

Responded  to 
Survey  2 

E-5  to  E-7 

3 

3 

2 

Captain  (0-3) 

3 

3 

3 

Major  (0-4) 

2 

2 

2 

Lt  Col  (0-5) 

3 

3 

1 

Civilian  (GS) 

3 

2 

2 

The  high  potential  for  rank  disparity  was  a  major  consideration  for  using  a  Delphi 


study  vice  a  moderated  panel.  Table  5  shows  the  range  of  ranks  considered  experts  and 
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therefore  qualified  as  panelists.  The  staff  and  WPS  perspectives  have  no  enlisted 
members  assigned  and  SPO  demographic  had  no  enlisted  members  available  for  the 
panel.  This  skewed  the  demographic  towards  officer  influence.  The  operator  perspective 
skewed  the  demographic  away  from  civilian  influence:  no  current  operators  are  federal 
civilians. 


Table  6:  Panel  Demographics — Crew  Position 


Accepted  Panel 

Responded  to 

Responded  to 

Crew  Position 

Invitation 

Survey  1 

Survey  2 

RPA  Pilot 

8 

8 

7 

RPA  Sensor  Operator 

3 

3 

2 

The  last  demographic  breakout,  Table  6,  shows  the  unavoidable  bias  towards  pilot 
perspective.  Like  the  rank  demographic,  the  staff  and  WPS  perspectives  required  pilots 
due  to  the  requirements  for  a  military  member  in  those  positions  to  be  an  officer  and 
therefore  not  a  sensor  operator.  One  out  of  three  panelists  from  both  the  operational  units 
and  the  TES  were  sensor  operators. 

Consensus  Area  Summary 

The  goal  of  the  surveys  is  to  form  consensus  among  the  panelists.  Since  Survey  1 
is  typically  open-ended,  any  consensus  areas  must  be  spontaneous  and,  in  this  case,  only 
covers  a  rather  broad  statement.  Specific  consensus  areas  are  listed  in  the  sections  below. 

Survey  1  Analysis 

There  was  one  area  of  consensus  in  Survey  1 :  users  have  an  active  role  in  the 
requirements  generation  process.  Of  the  thirteen  responses  to  Survey  1,  twelve  members 
(92%)  stated  that  operators  should  have  an  active  role  in  the  system  upgrade  or 
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modification  process  as  shown  below  in  Table  7.  Panelist  Golf  stated  that  operators  are 
“important”  to  the  process,  but  did  not  specify  the  specific  role  of  operators.  More  details 
are  discussed  below  in  the  section  labeled  “A  feedback  program  should  exist.” 


Table  7:  Survey  1  Consensus  Summary 


Statement 

Agreement 

1.  What  is  the  user's  role  in  the  system  upgrade  or  modification  process? 

Users  should  have  an  active  role 

92% 

The  open-ended  responses  about  feedback  systems  in  Survey  1  indicated  a 
potential  need  for  any  feedback  system  to  be  timely  as  seven  of  the  thirteen  respondents 
volunteered  “timely”  or  a  synonym  of  “timely”  as  a  top  characteristic.  Two  panelists 
stated  “advertisement”  as  a  desired  quality.  Other  volunteered  characteristics  had  less 
duplication,  but  included  transparency,  accountability,  clarity,  accuracy,  effectiveness, 
objectivity,  responsiveness,  and  simplicity. 

Survey  1  also  revealed  concerns  about  the  requirements  generation  process 
outside  of  feedback  systems.  Two  of  the  responses  dealt  with  executing  the  results  of  the 
feedback:  one  lamenting  the  high  cost  of  fixing  “things  that  get  missed,”  and  one 
recommending  open-architecture  to  decrease  cost  of  upgrades  through  competition.  An 
additional  three  responses  lamented  the  lack  of  expert-based  decisions — one  explicitly 
requesting  expert-based  or  analysis-based  leadership  decisions,  one  that  recommends  a 
warfighter’s  council  of  instructor  operators  and  maintenance  leads  to  review  contracts 
before  release,  and  one  that  laments  the  lack  of  expert  operators  available  to  advise  other 
stakeholders  during  the  early  developing  phase.  Another  panelist’s  comment 
recommends  use  of  Federally  Funded  Research  and  Development  Centers  (FFRDC), 
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industry  nodes,  Air  Force  Research  Labs  (AFRL),  and  Defense  Advanced  Research 
Projects  Agency  (DARPA)  to  facilitate  data-driven  analysis  of  technical  solutions  to  a 
tactical  problem.  Other  responses  included  a  request  for  streamlined  transparency;  agile 
development  methods;  modular  systems  design;  open-architecture  standards;  increased 
adherence  to  human-machine  interface  standards;  clear  identification  of  high-level 
stakeholders;  proper  distinction  between  the  Combatant  Commands  (COCOM)  setting 
the  required  mission  effect  and  MAJCOMs  determining  the  derivative  system 
requirements;  and  streamlined  but  consistent  fielding  processes. 

Survey  2  Analysis 

The  ten  respondents  to  Survey  2  answered  all  of  the  questions.  Table  8  and  Table 
9  summarize  the  ten  responses  to  all  five  questions,  separated  into  consensus  areas  and 
non-consensus  areas.  Figure  3  summarizes  the  responses  from  Question  5. 


Table  8:  Survey  2  Consensus  Summary 


Statement 

Agreement 

1.  User  Involvement:  What  types  of  user  involvement  should  be  accepted? 

The  Air  Force  should  NOT  seek  unfiltered,  unsolicited  feedback 

80% 

The  Air  Force  should  NOT  seek  feedback  vetted  only  through 
commanders 

70% 

2a.  Who  should  be  the  final  approval  authority  for  change  requests? 

The  final  decision  authority  SHOULD  be  the  Lead  MAJCOM 

70% 

The  final  decision  authority  should  NOT  be  the  "Lead"  COCOM 

90% 

The  final  decision  authority  should  NOT  be  a  Council  Vote 

80% 

2b.  Who  of  the  following  should  have  influence  or  give  suggestions  about  the  change 
request? 

HAF  should  NOT  influence  change  requests 

80% 

The  lead  MAJCOM  SHOULD  influence  change  requests 

100% 

Any  MAJCOM  SHOULD  influence  change  requests 

70% 
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The  "lead"  COCOM  should  NOT  influence  change  requests 

80% 

Any  COCOM  should  NOT  influence  change  requests 

80% 

The  Test  &  Evaluation  Squadron  commander  SHOULD  influence  change 
requests 

80% 

3.  What  review  process  would  best  facilitate  timely  and  complete  reviews  of 
submissions? 

A  group  discussion  with  user,  TES,  decision  maker's  staff  is  NOT  the  best 
review 

100% 

4.  What  data  architecture  would  BEST  assist  good  communication  without  too  much 
communication? 

A  pull  system  (i.e.  individuals  must  periodically  check  the  library  for 
relevant  submissions)  is  NOT  the  best  method. 

100% 

A  push  and  a  pull  system  (i.e.  individuals  may  both  seek  out  relevant 
submissions  and  be  automatically  notified  when  submissions  of  user- 
defined  criteria  are  created)  is  the  BEST  method. 

70% 

A  limited-access  online  library  where  only  trusted  agents  (i.e.  not  line 
operators,  just  the  change-request  processors)  have  access  is  NOT  the 
best  method. 

70% 

5.  Relative  Importance  of  specific  items  of  a  change  request.  Do  you  agree  with  the 
average  ranking  (+/- 1  positions)? 

Attachments 

70% 

Temporary  solution 

70% 

Reviewer  comments 

80% 

Reviewer's  information 

80% 

Table  8  provides  a  summary  of  Round  2  responses  that  show  consensus  in  the 
participants;  defined  as  70%  or  higher  agreement.  Nineteen  of  thirty-seven  responses  had 
partial  to  full  consensus,  as  reflected  with  green  shading.  Three  responses  had 
unanimous  agreement  and  are  shaded  blue.  Narrowing  the  focus  on  Questions  1-4 
reveals  that  ten  of  fifteen  consensus  areas — i.e.  two-thirds — rejected  rather  than  affirming 
a  particular  statement.  Essentially  the  panel  typically  knew  what  was  not  desirable  rather 
than  what  was  desirable. 
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Average  Importance  Ranking 

Agreement=%  of  responses  within  1  position  of  the  average  response 


Figure  3:  Average  Importance  Ranking  For  Specific  Data  Fields 


Figure  3  shows  both  the  average  results  for  the  relative  importance  of  specific 
feedback  items  for  a  change  request.  Agreement  for  this  question  is  the  percentage  of 
responses  that  are  within  one  position  of  the  entire  panel’s  average.  Columns  shaded  red 
have  less  than  70%  agreement;  columns  shaded  green  have  70%  or  more  agreement. 


Table  9:  Survey  2  Non-consensus  Summary 


Statement 

Agreement 

1.  User  Involvement:  What  user  types  of  user  involvement  should  be  acce 

pted? 

The  Air  Force  SHOULD  seek  Feedback  vetted  through  functionals 

50% 

The  Air  Force  SHOULD  seek  Feedback  through  formal  studies  or 
currently  established  feedback  systems 

60% 
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2b.  Who  of  the  following  should  have  influence  or  give  suggestions  about  the  change 
request? 

The  weapons  school  commander  SHOULD  influence  change  requests 

60% 

Any  commander  SHOULD  influence  change  requests 

50% 

3.  What  review  process  would  best  facilitate  timely  and  complete  reviews  of 
submissions? 

The  best  review  is  a  sequential  flow:  User-^Squadron  Weapons  and 
Tactics->Group  Weapons  &  Tactics-^decision  maker 

20% 

The  best  review  is  a  sequential  flow:  User-^Squadron  Weapons  and 
Tactics->Group  Weapons  &  Tactics -►Operational  Test  &  Eval 
Squadron-^decision  maker 

20% 

The  best  review  is  a  sequential  flow:  User-^Operational  Test  &  Eval 
Squadron-^decision  maker 

20% 

The  best  review  is  a  group  discussion  with  user,  SPO,  decision  maker's 
staff 

20% 

The  best  review  is  a  group  discussion  not  listed  on  the  survey 

20% 

5.  Relative  Importance  of  specific  items  of  a  change  request.  Do  you  agree  with  the 
average  ranking  (+/- 1  position)? 

Sub-system 

60% 

Phase  of  Flight 

60% 

Justification 

60% 

Proposed  Solution 

50% 

Urgency 

20% 

Geographic  Area  of  Operations 

30% 

Submitter's  information 

20% 

User-accessed  feasibility 

30% 

User-assessed  negative  aspects 

60% 

Analysis  Framework 

There  were  two  goals  of  survey  analysis.  The  first  goal  is  to  determine  the 
correlation  between  the  panelist’s  responses  and  the  ideal  characteristics  discovered 
during  the  literature  review.  The  second  was  to  define  specific  methods  to  attain  two  of 
the  attributes:  clear  methodology  and  capturing  complete  requirements.  For  the  first  goal, 
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Survey  2  explicitly  confirmed  six  characteristics  and  rejected  a  characteristic  but  was 
unable  to  confirm  or  reject  three  characteristics.  Clear  methodology  had  four  parts 
corresponding  to  the  first  four  questions  of  Survey  2.  Two  of  these  questions  had 
significant  consensus;  two  did  not.  The  definition  of  complete  requirement  capture 
started  in  Survey  1  and  was  expanded  in  Survey  2.  Four  of  the  thirteen  individual 
feedback  items  had  consensus  for  their  relative  rank  on  an  importance  scale.  Of  those 
four,  the  rank  of  two  items  has  significance  for  implementing  a  program.  A  summary  of 
these  results  is  below. 


Table  10:  Panel  Assessment  of  Ideal  Characteristics 


Ideal  Characteristic  Identified  in  Chapter  2 

Panel  Assessment 

A  feedback  program  should  exist 

Confirmed 

Focuses  on  refining  current  functionality 

Confirmed 

Focuses  feedback  on  actionable  subjects 

Confirmed 

Accommodates  various  feedback  mediums 

Confirmed 

Captures  user  context/environment 

Confirmed 

Initiators  trust  inputs  have  impact 

Confirmed 

Highlights  unintended  consequences 

Rejected 

Low  user  effort  required 

No  assessment 

Automated  triage  of  feedback 

No  assessment 

Captures  consistent  requirements 

No  assessment 

Captures  complete  requirements 

Further  explored 

Clear  methodology 

Further  explored 

Additionally,  the  responses  from  panelists  that  have  staff  or  Test  &  Evaluation 


experience  were  separated  to  look  for  significant  differences  in  demographics. 

Responses  from  the  Weapon  School  and  System  Program  Office  perspective  were  not 
independently  evaluated  due  to  low  turnout  in  those  areas.  The  Operator  perspective  was 
not  independently  evaluated  since  all  but  one  Survey  2  respondent  has  operator 
experience.  For  Questions  1-4,  significant  deviation  from  the  whole  panel  is  defined  as 
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exceeding  half  a  vote  divided  by  the  number  in  the  demographic.  For  example,  if  50%  of 
the  whole  panel  agreed  on  a  question  and  2  of  3  Staff  panelists  agreed,  the  deviation  is 
not  significant  (50%  is  within  1/6  of  67%).  Conversely,  if  70%  of  the  panel  agreed  and 
only  2  of  4  Test  &  Evaluation  panelists  agreed,  then  the  deviation  is  significant.  For 
Question  5,  significance  is  defined  as  an  item  whose  rank  moved  three  or  more  positions. 
Confirmed  ideal  characteristics 

The  panelist’s  desired  characteristics  have  significant  overlap  with  the  ideal 
characteristics  discussed  in  Chapter  2.  The  sections  below  describe  the  areas  of  overlap. 

A  feedback  program  should  exist 

Twelve  of  thirteen  respondents  in  survey  1  indicated  that  operators  should  be 
involved  in  the  requirements  process.  Panelist  Golf  indicated  that  operators  are  important 
in  the  process  of  system  modification,  but  did  not  say  that  the  operators  should  be  directly 
involved.  Nine  of  the  responses  explicitly  stated  operators  should  give  feedback  on  the 
functionality  of  the  system.  The  remaining  three  responses  discussed  the  need  for 
operator  involvement  during  requirements  development,  but  did  not  explicitly  discuss 
feedback. 

Focuses  on  refining  current  functionality 

In  Survey  1,  a  majority  of  respondents  identified  the  need  to  capture  a  proposed 
solution  to  particular  problem  and/or  the  current  effectiveness  of  the  system.  In  Survey  2, 
the  respondents  ranked  two  individual  feedback  items  for  usefulness:  the  proposed 
solution  and  justification.  Both  of  these  items  seek  to  improve  the  function  of  a  system. 
The  proposed  solution  describes  a  potential  fix  to  a  problem  and  was  ranked  number  4  of 
13.  The  justification  describes  the  problem  as  it  relates  to  either  safety  or  mission  and 
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was  ranked  number  3  of  13.  Of  note,  the  staff  demographic  ranked  the  proposed  solution 
as  number  1 . 

Focuses  feedback  on  actionable  subjects 

The  environment  this  feedback  system  would  exist  is  already  somewhat  focused. 
First,  the  feedback  is  limited  to  only  materiel  problems.  Concerns  with  non-materiel 
problems,  such  as  tactics  improvements,  are  captured  via  the  processes  described  in 
Chapter  2.  Second,  the  feedback  is  focused  on  a  single  aircraft  series.  Even  still,  the 
panel  did  confirm  two  ways  to  focus  feedback. 

Focusing  feedback  has  two  distinct  subordinate  concepts:  identifying  a  feedback 
item  that  can  focus  feedback  and  allow  users  to  see  the  areas  that  other  users  have 
provided  feedback.  Both  of  these  concepts  were  confirmed  in  Survey  2.  First,  the 
average  ranking  for  the  “sub-system”  feedback  item  was  first  place  meaning  it  was  the 
single  most  important  piece  of  information  presented  in  the  survey.  Second,  the  panel 
rejected  the  “trusted  agent”  data-architecture  option  and  confirmed  the  “push  and  pull” 
method  of  disseminating  previous  feedback  areas.  Disseminating  the  approved  areas  of 
feedback  is  a  key  piece  to  focusing  feedback  towards  actionable  areas. 

Accommodates  various  feedback  mediums 

This  characteristic  was  the  genesis  for  the  potential  feedback  item  of 
“Attachments  to  support  videos,  pictures,  or  other  non-textual  feedback”  option  for 
Question  5  of  Survey  2.  This  feedback  item  was  not  mentioned  in  Survey  1,  however  it 
received  a  strong  ranking:  sixth  out  of  thirteen.  Additionally,  it  was  one  of  the  few  items 
that  was  consistently  ranked.  Seven  of  the  ten  respondents  ranked  “attachments” 
between  fifth  and  seventh  in  importance. 
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Captures  user  context  and  environment 

This  characteristic  also  drove  two  individual  feedback  items  for  Question  5  of 
Survey  2:  “Phase  of  flight,”  and  “Geographic  area.”  These  two  items  are  an  adaptation  of 
the  context  captured  via  the  Contexter  feedback  tool.  The  Contexter  tool  also  relied  on 
user’s  profile  infonnation,  however  “Submitter’s  information”  was  already  a  part  of  the 
potential  list  of  feedback  items.  Two  items — Geographic  Area  and  Submitter’s 
Infonnation — ranked  somewhat  low  on  Survey  2  scoring  an  average  of  eight  and  nine  out 
of  thirteen  respectively.  “Phase  of  Flight,”  however,  scored  extremely  well  at  second  of 
thirteen.  Additionally,  the  Test  &  Evaluation  demographic  also  ranked  “Phase  of  Flight” 
as  number  two.  This  breakout  is  significant  because  the  Test  &  Evaluation  Squadron’s 
own  feedback  tool  does  not  directly  record  this  item. 

Initiators  trust  inputs  have  impact 

The  panelists  indirectly  confirmed  the  need  to  trust  the  feedback  system  in  Survey 
2.  Starting  in  Survey  1,  roughly  half  of  the  panelists  remarked  that  an  ideal  feedback 
system  needed  good  communication  via  either  transparency,  accountability,  advertising 
the  results,  or  clarity.  In  Question  4  of  Survey  2,  the  panelists  confirmed  the  ideal  data 
repository  to  increase  communication  and  therefore  increase  the  potential  for  trust. 
Rejected  ideal  characteristics 

The  panel  rejected  one  of  the  ideal  characteristics  discussed  in  Chapter  2.  The 
section  below  describes  how  that  characteristic  does  not  apply  to  the  RPA  community. 

Highlights  unintended  consequences 

This  characteristic  was  introduced  as  an  option  to  include  “User-assessed  negative 
aspects  to  the  change  request”  as  a  specific  area  of  feedback.  This  specific  feedback  item 
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was  rated  as  rather  unimportant  in  Survey  2(11  of  13).  This  is  most  likely  due  to  the 
differing  roles  the  panelists  perceived  in  the  feedback  process.  Operators  may  understand 
the  negative  impacts  when  requesting  a  change  to  the  interface;  however,  operators 
expect  engineering  reviews  to  detennine  any  negative  consequences.  This  is  reflected  in 
the  panel’s  low  ranking  of  user  assessments. 

Characteristics  neither  confirmed  nor  rejected 

The  panel  did  not  explicitly  confirm  nor  deny  any  of  the  following  ideal 
characteristics.  Generally  speaking,  the  panel  did  not  volunteer  the  following  areas  as 
areas  of  concern  during  Survey  1 . 

Low  user  effort  required 

This  characteristic  was  not  fully  investigated  because  the  panel  did  not  explicitly 
state  a  desire  for  an  easy  feedback  method.  The  literature  review  discussed  how  to 
decrease  the  amount  of  effort  required  to  submit  feedback  and  specifically  recommended 
a  semi-automated  smart-phone  application.  That  specific  solution  is  not  practical  for 
military  aviation — the  simplest  solution  is  to  fill  out  a  fonn  and  email  it  straight  to  the 
appropriate  point  of  contact  (POC).  The  key  element  of  this  “simplest”  solution — the 
direct  communication  with  the  POC — was  presented  to  and  rejected  by  the  panel  (80% 
did  not  recommend  direct,  unfiltered  communication).  That  said,  other  methods  of 
easing  the  difficulty  of  sending  feedback  were  not  investigated. 

Automated  triage  of  feedback 

In  addition  to  the  previous  characteristic,  automated  review  of  feedback  was  not 
mentioned  in  the  surveys.  The  core  of  this  characteristic  is  a  time  saving  method  for 
reviewing  large  numbers  of  individual  reviews.  This  concern  was  reflected  in  Panelist 
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Hotel’s  comments  on  Survey  2:  “having  a  vetting  process  locally  will  help  MAJCOM 
and  staff  focus  the  limited  resources.”  Fitting  with  this  sentiment,  the  Panel  assessed 
multiple  potential  vetting  process  flows  but  no  automated  flows  were  evaluated. 

The  broad  desire  for  human  review  over  automatic  review  is  not  surprising  given 
the  difference  between  the  intent  behind  the  military  and  commercial  feedback  process. 
The  literature  review  discussed  methods  to  condense  a  multitude  of  online  reviews  into 
the  key  features.  However,  the  original  purpose  of  an  online  review  is  to  advise  any 
potential  buyers  in  making  the  decision  between  buying  one  out  of  a  group  of  similar 
products.  Operator  feedback  in  the  military  is  focused  on  comparing  a  single  product, 
often  the  only  one  of  its  type,  to  the  stated  mission.  This  feedback  typically  aims  to 
improve  the  system  and  influences  only  one  final  decision  authority.  The  differences  in 
the  number  of  decision  makers  and  the  purpose  of  feedback  creates  a  disconnect  between 
the  military’s  and  industry’s  desire  for  automated  triage. 

Captures  consistent  requirements 

The  consistency  of  requirements  was  not  heavily  investigated  in  the  surveys.  The 
intent  behind  this  characteristic  is  to  ensure  that  one  requirement  does  not  conflict  with 
another  requirement.  For  change  requests,  an  inconsistency  can  occur  between  the 
proposed  change  and  previous  requirement.  The  specific  feedback  item  of  “User 
assessed  negative  aspects  to  the  change  request”  was  the  only  aspect  of  this  characteristic 
investigated.  The  full  application  of  the  heuristics  described  in  the  literature  review  is 
best  suited  for  a  discussion  of  technical  reviews  of  change  requests,  but  that  aspect  was 
not  fully  examined. 
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Program  Implementation  Specifics 

The  following  sections  describe  the  results  from  the  panel’s  expanded 
investigation  into  two  specific  characteristics  of  a  good  feedback  program:  clear 
methodology  and  capturing  complete  requirements. 

Clear  methodology 

This  characteristic  surfaced  in  Survey  1  and  further  developed  in  Survey  2. 
Panelist  Alpha  and  Delta  both  stated  transparency  was  an  ideal  characteristic.  Panelist 
India  desired  a  clear  goal  and  a  clear  process  and  Panelist  Echo  had  this  observation: 
“Eve  been  told  the  process  [of  providing  feedback]  is  alive,  but  rarely  used  because  no 
one  understands  it.”  Questions  1-4  of  Survey  2  sought  to  define  the  exact  methodology 
in  relation  to  this  characteristic. 

The  first  four  questions  of  Survey  2  roughly  correlated  with  the  four  broad  steps 
SWEBOK  recommends  for  software  change  requests.  How  to  originate  the  request  is 
tied  to  Question  1.  The  review  flow  was  investigated  in  Question  3.  The  method  and 
position  to  make  the  decision  was  examined  in  Question  2.  Finally,  a  way  to  report  the 
final  detennination  was  explored  in  Question  4.  In  all,  the  panel  had  two  areas  of 
constructive  consensus:  the  final  decision  authority  and  the  best  database  were  both 
selected.  The  other  two  areas — who  should  initiate  a  change  request  and  how  should  the 
request  be  reviewed — only  had  consensus  for  options  not  to  pursue.  The  following 
paragraphs  describe  each  area  in  detail. 

Formal  Change  Request  Originators 

In  Question  1  the  panel  did  not  confirm  who  should  be  the  proper  originators  of 
formal  change  requests.  The  panel  did  have  consensus  rejecting  the  utility  of  both 
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unfiltered,  unsolicited  feedback  (80%)  and  feedback  only  through  commanders  (70%). 
The  panel  was  neutral  on  the  utility  of  channeling  feedback  through  functionals  and 
through  current  feedback  methods  such  as  the  feedback  methods  discussed  in  literature 
review.  Additionally,  two  panelists  recommended  other  solutions.  Panelist  Bravo 
recommended  a  panel  of  experts  similar  to  the  Delphi  method  employed  for  this  research 
and  Panelist  Echo  recommended  the  solution  was  to  educate  more  operators  in  the 
current  methods  for  feedback.  The  demographic  breakout  of  TES  panelists  meets 
consensus  for  all  four  statements  at  75%  rejection  of  unfiltered  and  commander-only 
feedback  and  75%  acceptance  of  current  processes  and  feedback  through  a  functional 
chain.  The  staff  demographic  is  also  significant:  unanimous  rejection  of  unfiltered 
feedback. 

Change  Request  Review  Process 

In  Question  3,  the  panel  assessed  potential  review  flows  that  would  filter  and  vet. 
From  Survey  1  analysis,  operators  are  familiar  with  the  sequential  review  process  of  the 
Form  847.  Other  comments  highlighted  two  potential  routes  for  review:  the  Test  & 
Evaluation  Squadron  and  the  functional  chain  inside  squadron-level  and  group-level 
Weapons  &  Tactics  shops.  The  group  choices  are  military  translations  of  the 
OpenProposal  group  discussions  between  users,  requirements  analysts,  and  engineers. 

This  question  had  the  least  consensus  implying  future  feedback  system 
development  should  consider  the  broad  range  of  opinions.  Of  the  ten  responses,  they 
were  evenly  spread  out  between  five  choices.  The  only  consensus  was  the  rejection  of  a 
group  discussion  between  the  operator,  the  Test  &  Evaluation  squadron  and  the  decision 
maker.  The  other  selections  mainly  show  individual  biases  either  for  a  known  process  or 
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a  fear  of  an  opposing  bias.  For  example,  Panelist  Charlie  was  involved  in  a  configuration 
working  group  with  heavy  involvement  with  the  SPO.  He  selected  the  group  discussion 
with  user,  SPO  and  decision  maker.  Another  example  is  Panelist  Echo,  a  Weapon  School 
Graduate  familiar  with  the  Tactics  Improvement  Program  (TIP).  He  selected  “other”  and 
described  a  flow  similar  to  the  TIP  flow:  a  sequential  flow  of  User->  Squadron  Weapons 
&  Tactics->Group  Weapons  &  Tactics->MDS-specific  weapons  school  followed  by  a 
group  discussion  between  the  Weapon  School,  the  TES,  the  SPO  and  the  decision 
maker’s  staff.  Panelist  Kilo,  a  TES-experienced  panelist,  also  selected  “other”  and 
described  an  initiative  inside  the  TES  to  incorporate  more  operator  involvement  in 
Deficiency  Reporting  through  Squadron  Weapons  &  Tactics  officers. 

The  three  sequential  flows  attempt  to  balance  the  length  of  the  review  chain  while 
balancing  any  potential  bias  from  sub-cultures.  The  first  flow  contains  only  Weapons  & 
Tactics  reviewers  who  may  be  biased  towards  kinetic  operations  over  reconnaissance 
operations  or  the  administration  of  takeoff,  cruise,  and  landing  operations.  The  second 
flow  includes  TES  in  the  review  flow  to  balance  the  Weapons  &  Tactics  influence.  This 
flow  is  slightly  longer  than  the  others  listed.  The  third  and  final  sequential  flow  includes 
only  TES  review.  As  discovered  during  the  surveys,  this  third  flow  is  currently  available; 
however,  it  is  not  well  advertised.  A  major  limiting  factor  is  the  TES’s  lack  of  presence 
in  line  squadrons  compared  to  other  agencies  gathering  feedback.  Line  operators  at  bases 
other  than  Creech  AFB — the  location  of  the  TES  squadron — may  only  see  a  single  TES 
member  during  major  upgrade  roadshows  occurring  every  few  years.  Other  feedback 
options  include  imbedded  representatives.  The  Form  847,  AF1067,  AFT022,  WEPTAC 
and  TIP  feedback  programs  all  have  squadron-level  shops  collecting  the  initial  feedback. 
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Final  Decision  Authority  For  Change  Requests 

In  Question  2,  the  panel  evaluated  who  should  be  the  final  decision  authority  for 
change  requests  and  what  other  entities  should  influence  that  decision  authority.  This 
area  showed  the  most  consensus  of  any  question  and  largely  followed  doctrinal  premises. 
70%  of  the  panel  said  the  Lead  MAJCOM  commander  should  be  the  final  decision 
authority.  Of  the  two  panelists  selecting  a  council  vote,  both  selected  all  MAJCOMs  but 
only  one  selected  COCOMs  as  having  a  vote.  The  final  panelist  selected  the  COCOM 
that  has  Operational  Control  (OPCON)  over  the  most  number  of  a  particular  MDS.  For 
influencing,  all  the  panelists  agreed  that  the  lead  MAJCOM  should  either  make  the 
decision,  be  on  the  council  or  should  influence  the  final  decision.  Additionally,  the  panel 
had  consensus  that  the  TES  commander  should  influence  decision.  Additional  comments 
from  Panelist  Kilo  align  with  the  entire  panel’s  perception  of  the  TES  role:  “the  role  of 
TES  is  to  represent  the  operator.”  Conversely,  the  panel  rejected  statements  for  any 
COCOM  commander  or  Headquarters  Air  Force  personnel  influencing  the  decision. 
Comments  from  Survey  1  indicate  the  role  of  COCOMs  and  HAF  is  to  set  the  desired 
mission  effect,  not  the  method  to  attain  it.  The  influence  of  any  commander,  including 
the  Weapon  School  commander,  did  not  reach  consensus. 

This  question  had  some  significant  demographic  breakouts.  All  four  TES- 
experienced  panelists  opposed  HAF  influence.  Additionally,  the  three  staff  members 
were  unanimously  in  favor  of  any  MAJCOM  influence  and  unanimously  opposed  to  HAF 
or  COCOM  influence. 
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Change  Request  Status  Dissemination 

In  Question  4,  the  panel  assessed  the  utility  of  various  data  architectures  and  their 
ability  to  aid  proper  communication.  The  panel  had  70%  consensus  that  a  “push  and 
pull”  database  is  the  appropriate  data  repository.  A  “pull”  database  could  be  as  simple  as 
an  online  library.  The  library’s  owner  adds,  changes,  or  deletes  infonnation  and  library 
users  must  remember  to  access  the  library  periodically  to  check  for  those  updates.  In 
contrast  a  “push  and  pull”  database  allows  the  former  but  also  sends  library  users  a 
notification  when  new  or  changed  information  is  available;  i.e.  it  pushes  the  information 
out  to  the  library  users.  The  user  can  typically  customize  settings  to  only  receive  certain 
categories  of  updates.  An  example  of  a  custom  filter  would  be  an  MQ-1  operator  looking 
for  notification  of  any  new  feedback  about  the  Ground  Control  Station  (a  common 
component  between  MQ- 1  and  MQ-9)  and  the  MQ- 1  airframe  but  not  the  MQ-9 
airframe.  The  utility  of  a  push-and-pull  database  is  to  efficiently  keep  interested  parties 
aware  of  all  available  feedback  items  about  a  certain  subject. 

The  remaining  panelists  assessed  the  best  data  repository  was  a  “trusted  agent” 
model  where  only  selected  trusted  users  have  access  to  the  full  reports.  Both  the  current 
library  for  Deficiency  Reports  and  Air  Force  Safety  Automated  System  (AFSAS)  only 
allow  access  for  selected  members  (“JDRS  Homepage,”  2015;  “AFSAS  Help,”  2015). 
One  reason  for  using  a  trusted  agent  model  is  to  limit  the  ability  for  infonnation  to  spread 
beyond  the  original  intent.  For  AFSAS,  this  concern  is  amplified  due  large  portions  of 
“Safety  Privilege”  information.  This  concern,  however,  did  not  deter  the  panel  from 
confirming  the  best  solution  was  not  the  trusted  agent  model. 
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The  final  option,  a  pull-only  database,  was  unanimously  rejected.  While  this 
method  is  the  easiest  to  implement,  it  has  neither  the  data  protections  found  in  the  trusted- 
agent  database  nor  the  ease  of  communication  found  in  the  push-and-pull  database. 

Capturing  complete  requirements 

Between  Question  8  of  Survey  1  and  Question  5  of  Survey  2,  the  panel  first 
identified  then  prioritized  thirteen  individual  feedback  items  that  should  be  included  in  a 
change  request.  These  items  are  listed  in  Figure  3  in  order  of  their  final  importance 
ranking;  i.e.  lower  ranked  items  assisted  communication  better  than  higher  ranked  items. 
Consensus  for  this  list  was  the  number  of  panelists  who  ranked  an  individual  item  within 
one  position  of  the  average  ranking.  There  were  only  four  items  that  had  consensus: 
attachments,  temporary  solutions,  reviewer  comments,  and  reviewer’s  information. 
Attachments  and  temporary  solutions  were  moderately  ranked  (6  and  7  out  of  13, 
respectively)  and  had  70%  consensus  for  both.  The  consensus  means  that  few  panelists 
considered  these  items  extremely  important  or  unimportant  and  should  be  included  but 
not  necessarily  emphasized  on  a  feedback  report.  The  other  two,  reviewer  comments  and 
information,  were  consistently  ranked  at  the  bottom  with  80%  consensus  for  both.  The 
consistent  poor  ranking  means  the  panel  is  more  interested  in  having  the  review  (see 
Question  1)  than  the  infonnation  uncovered  during  the  review.  This  also  implies  the 
panel  is  confident  that  reviewers  will  reject  inadequate  change  requests  rather  than 
writing  comments  and  passing  the  change  request  along.  These  two  items  should  be 
relatively  minor  during  feedback. 

The  lack  of  consensus  for  the  other  nine  items  is  also  significant.  Essentially 
these  items  had  large  variation  among  the  panel  on  the  relative  importance.  Fortunately 
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consensus  is  not  required  as  the  individual  items  are  not  mutually  exclusive.  At  a 
minimum,  the  relative  rank  does  identify  which  areas  should  get  the  most  attention  while 
capturing  the  feedback. 

The  staff  demographic  had  significant  differences  when  compared  with  the  entire 
panel.  Specifically  the  staff  valued  the  proposed  solution  as  the  most  important  (#1) 
feedback  item,  up  from  #4  for  the  whole  panel.  Additionally,  the  staff  valued  the  user’s 
assessment  significantly  more  than  the  whole  panel,  ranking  it  #3  up  from  #10.  Finally, 
the  staff  was  less  concerned  about  the  estimated  urgency  of  the  change  request,  ranking  it 
#11  down  from  #5.  There  are  no  additional  comments  directly  from  the  panel  detailing 
any  motivations  for  the  differences;  however,  the  staff  may  be  more  concerned  about 
delineating  between  what  can  be  done  vice  what  cannot  and  less  concerned  about  the 
distinction  between  the  short-term  than  the  long-tenn  timeline. 

The  TES  viewpoint  confirmed  that  the  deficiency  report  process  already  collects 
much  infonnation  while  generating  change  requests.  Panelist  Kilo  included  an  example 
Deficiency  Report  with  his  survey  responses.  The  survey  responses  showed  some 
significant  overlap  and  significant  differences  between  the  survey  responses  and  the 
current  Deficiency  Report. 

Table  1 1  below  summarizes  the  comparison.  Of  note,  the  TES  squadron  is  not 
currently  capturing  the  phase  of  flight  independently  from  the  problem  description 
although  the  both  TES-only  demographic  and  the  panel  as  a  whole  ranked  this  item  as 
number  2. 


Table  11:  Comparison  of  Deficiency  Report  and  Survey  Results 
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Captured  only  in  the 
Deficiency  Report 
template 

Captured  in  both 
Deficiency  Reports  and 
the  Delphi  Survey 
(#  =  survey  rank) 

Captured  only  in  the 
Delphi  Survey 
(#  =  survey  rank) 

Hazard  Priority 

Sub-system  (1) 

Phase  of  Flight  (2) 

Problem  Description 

Justification  (3) 

Estimated  Urgency  (5) 

Known  Similar 

Recommendation  / 

Attachments  (6) 

deficiencies 

Proposed  Solution  (4) 

How  was  the  problem 
detected 

Submitter's  information 
(9) 

Temporary  solution  (7) 

Specific  version 

Geographic  Area  of 

numbers  of  subsystems 

Operations  (8) 

User-accessed  feasibility 
(10) 

User-assessed  negative 
aspects  (11) 

Reviewer  comments 
(12) 

Reviewer's  information 
(13) 

Summary 

This  chapter  summarized  and  interpreted  the  survey  responses  oriented  towards 
answering  two  questions:  should  the  Air  Force  should  establish  a  program  to  collect 
unsolicited  operator  feedback,  and  if  so,  how  should  the  program  operate.  The  actual 
response  demographics  and  metrics  detailed  the  constitution  of  the  panel.  The  areas  of 
consensus  and  non-consensus  allows  for  a  confirmation  of  six  characteristics  and  the 
rejection  of  an  additional  characteristic.  Finally  the  panel  assessed  two  specific 
characteristics  in  greater  detail:  clear  methodology  and  capturing  complete  requirements. 
The  panel  was  able  to  meet  consensus  on  just  under  half  of  sub-areas  required  for 
program  implementation.  The  program  implementation  consensus  areas  tie  directly  to 
the  conclusions  and  recommendation  in  the  next  chapter. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

The  Air  Force  already  runs  several  feedback  systems;  however  no  one  system 
meets  all  the  characteristics  the  panel  desired  or  that  were  identified  in  the  literature  as 
key  characteristics.  One  current  military  program  (Deficiency  Reports)  and  one 
commercial  tool  (Open  Proposal)  show  the  most  promise  for  meeting  all  of  the  factor 
characteristics.  This  research  recommends  specific  improvements  to  the  current  system 
and  an  application  of  the  tool  to  another  feedback  program.  Additionally,  there  are 
aspects  of  feedback  systems  this  thesis  did  not  cover  and  aspects  of  the  whole  acquisition 
system  that  surfaced  while  discussing  feedback  systems.  Those  two  conditions  are 
detailed  below  as  recommended  future  research. 

Conclusions  of  Research 

The  research  has  answered  the  two  investigative  questions  from  Chapter  1 :  should 
the  Air  Force  establish  an  unsolicited-feedback  program  for  line  operators,  and  if  so, 
what  should  the  basic  characteristics  be?  In  Chapter  2,  selected  feedback  systems 
currently  in  use  were  compared  to  literature-generated  ideal  characteristics.  Through  the 
methodology  in  Chapter  3,  the  expert  panel  assessed  the  characteristics  with  the  results 
presented  in  Chapter  4.  Below  is  Table  12,  an  edited  copy  of  Table  1  only  displaying 
characteristics  the  panel  confirmed. 
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Table  12:  Filtered  Summary  of  Current  Feedback  System  Critiques 


Available  to  /  designed  for  any  line  operator 

Advertised  to  all  line  operators 

Focuses  on  refining  current  functionality 

Captures  complete  requirements 

Clear  methodology 

Accommodates  various  feedback  mediums 

Captures  user  context/environment 

Focuses  feedback  on  actionable  subjects 

Initiators  trust  inputs  have  impact 

AF1067 

X 

P‘ 

X 

P2 

P3 

X 

AF847 

X 

X 

X 

P2 

P3 

X 

AFTO  22 

X 

X 

X 

P2 

P3 

X 

JLLIS 

X 

X 

P3 

Deficiency  Reports 

X 

P9 

X 

X 

X 

P2 

P3 

p7 

Cockpit  Working  Grp 

9 

p 

X 

X 

X 

X 

ARC  WEPTAC 

p9 

X 

X 

X 

X 

TIP 

p9 

X 

X 

X 

X 

Sales  Representative 

X 

X 

X 

p8 

X 

P8 

NASA  Space  Shuttle 

X 

X 

X 

X 

X 

Idavall 

X 

X 

X 

X 

X 

X 

X 

Contexter 

X 

* 

X 

X 

X 

X 

Open  Proposal 

X 

* 

X 

X 

X 

X 

X 

X 

Data  mining 

* 

X 

X=fully  meets  the  characteristic 
p — parti  ally  meets  the  characteristic 
*=not  applicable 

p1  -  this  method  has  significant  limitations  on  changing  system  functionality 
p2  -  these  methods  allow  non-video  attachments 

p3  -  these  methods  have  free  text  entry  areas  that  may  include  context  if  the  initiator  is  aware  of 
the  importance  of  context 

p7  -  initiators  are  typically  have  database  access  as  they  are  not  line  operators 

p8  -  most  feedback  is  informal  and  relies  on  individual  skill-level 

p9  -  initiator  commonly  has  informal  contact  with  majority  of  unit-level  operators 
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The  re-scoped  summary  of  current  feedback  system  critiques  shows  three  systems 
that  meet  most  of  the  desired  characteristics.  The  OpenProposal  tool  and  IdavalTs 
feedback  program  are  still  highly  effective  programs  as  discussed  at  the  end  of  Chapter  2. 
IdavalTs  operating  construct  would  be  difficult  to  adapt  to  the  military.  IdavalTs  end- 
users  are  the  same  as  the  paying  customers  vice  end-users  do  not  make  the  decisions  in 
the  military.  Additionally,  IdavalTs  stakeholders  are  relatively  homogenous — a 
condition  not  true  for  the  military.  Conversely,  OpenProposaTs  strengths  and  basic 
design  potentially  correlate  to  the  concept  of  the  Cockpit  Working  Group.  OpenProposal 
was  designed  to  focus  a  group  of  distinct  roles  on  improving  the  functionality  of  a 
computer’s  interface.  Likewise,  the  CWG  pulls  various  roles  together  to  improve  the 
design  of  the  aircraft’s  interface:  the  cockpit.  The  third  program  is  the  Deficiency 
Reporting  program.  This  program  lacked  most  of  the  ideal  characteristics  the  board  was 
unable  to  confirm;  however,  it  accommodates  the  vast  majority  of  confirmed 
characteristics.  Evaluating  Deficiency  Reporting  against  the  desired  characteristics 
shows  the  program  completely  or  partially  meets  all  the  desired  characteristics  except 
one.  Additionally,  Deficiency  Reporting  process  follows  many  of  the  specific 
methodologies  confirmed  in  Survey  2.  The  recommendations  below  describe  methods  to 
improve  current  military  programs  and  potential  benefits  to  implementing  an  adaptation 
of  a  commercial  method  for  military  feedback. 

Significance  of  Research 

This  research  has  identified  desired  characteristics,  some  of  the  basic  architecture 
of  a  feedback  system  to  collect  unsolicited  operator  feedback,  and  critiqued  several 
feedback  programs.  The  Air  Force  has  highly  trained  crewmembers  that  are 
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underutilized  in  the  requirements  refinement  process.  While  not  all  of  the  investigative 
questions  were  completely  answered,  the  research  identified  best  practices  and 
highlighted  potential  changes  to  current  programs  that  may  allow  for  significant 
improvements  with  relatively  minor  amounts  of  cost  and  energy.  The  inclusion  of 
multiple  perspectives  broadened  and  combined  the  collective  knowledge  of  user-driven 
feedback  methodology  in  the  Air  Force.  Finally,  the  research  has  also  highlighted  areas 
where  more  investigation  may  be  warranted  to  further  improve  requirements  definition 
and  refinement  thereby  potentially  increasing  system  effectiveness. 

Recommendations  for  Action 

The  research  generated  five  specific  recommendations.  Some  of  these 
recommendations  involve  modifications  to  publications  or  technical  orders.  The  first 
four  recommendations  deal  with  specific  actions  to  expand  the  access,  advertisement,  and 
transparency  of  the  deficiency  reporting  process.  The  last  recommendation  is  a  potential 
improvement  for  Cockpit  Working  Groups. 

The  first  recommendation  stems  from  the  concept  of  both  focusing  feedback  and 
capturing  the  context  around  a  change  request:  add  “phase  of  flight”  to  the  standard 
deficiency  report  template.  The  two  goals  bound  the  proper  amount  of  selectable  phases 
on  both  ends.  Limiting  the  available  phases  to  “Launch  and  Recover  Element  (LRE)” 
and  Mission  Control  Element  (MCE)”  does  not  truly  capture  the  context — the  MCE 
environment  covers  all  phases  except  tenninal  phase  of  flight  and  is,  therefore,  overly 
broad.  Conversely,  listing  every  possible  phase  of  flight  creates  a  list  too  long  to  be 
reasonably  remembered  and  does  not  focus  the  feedback  to  specific  areas.  The  author 
recommends  the  following  phases:  ground  operations,  takeoff,  departure,  enroute, 


76 


mission,  arrival,  and  landing.  Additionally,  the  mission  phase  can  be  separated  into 
specific  mission  types;  i.e.  reconnaissance,  surface  attack  tactics,  Time-Sensitive  Target 
(TST)  attack,  Strike  Coordination  and  Reconnaissance  (SCAR),  Close  Air  Support  (CAS) 
or  Combat  Search  and  Rescue  (CSAR).  The  panel  confirmed  the  relative  importance  of 
this  particular  feedback  item  during  Survey  2. 

The  next  three  recommendations  aim  to  increase  the  transparency  of  and  operator 
involvement  in  the  DR  process.  The  first  of  these  is  semantic  in  nature:  add  an  explicit 
definition  of  a  deficiency  to  AFI  99-103  and  TO  00-35D-54.  Both  of  those  regulations 
dictate  procedures  for  categorizing  and  processing  deficiencies  but  neither  explicitly 
defines  them.  Lack  of  a  clear  definition  shrouds  the  scope  of  the  DR  process  in 
uncertainty  and  allows  for  operator  to  misperceive  limits  on  the  DR  program.  Combining 
the  definitions  for  the  distinct  types  of  deficiencies  implies  a  deficiency  is  any  material  or 
design  condition  that  is  unsafe  or  limits  the  use  of  the  material  for  the  purpose  intended 
due  to  material  defect,  design  defects,  or  specification  inadequacy  (TO  00-35D-54, 

201 1:C5).  Deficiencies  should  also  include  design  enhancements  that  complement  or 
improve  mission  suitability  or  effectiveness  even  if  incorporating  the  enhancement  is  not 
absolutely  required  for  successful  mission  accomplishment. 

Culturally  at  least  one  member  of  the  TES  squadron  considered  the  OT&E 
squadron  as  the  “voice  of  the  operator”  when  dealing  with  other  acquisition  stakeholders. 
Formally  capturing  in  doctrine  that  role  could  expand  the  coverage  of  the  operator’s 
viewpoint  for  all  current  feedback  methods  involving  OT&E.  Specifically,  AFI  99-103 
should  require  the  general  T&E  community  to  expand  the  responsibilities  from  “provide 
information  to  users”  to  also  include  “educate  on  current  feedback  methods”  and  “seek 
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operator’s  engagement  in  product  improvement”  (AFI  99-103,  2013:7).  More 
specifically,  this  responsibility  is  most  appropriate  for  the  Operational  T&E  elements  and 
should  be  clearly  stated  as  such  (AFI  99-103,  2013:29).  Additionally  the  rate  of  success 
for  reaching  and  engaging  operators  is  a  metric  worth  tracking.  Specifically  the 
percentage  of  DRs  originating  with  a  line  operator  should  be  added  to  the  TO  00-35D-54 
required  metrics  under  the  topic  of  “Warfighter  Satisfaction”  (TO  00-35D-54,  201 1:A1). 

The  last  recommendation  for  increasing  access  and  transparency  is  to  allow 
operators  access  to  the  database  library  for  DRs,  specifically  Joint  Deficiency  Reporting 
System  (JDRS)  (AFI  99-103,  2013:58).  For  sake  of  need-to-know,  the  access  can  be 
read-only  except  for  deficiency  submissions  and  limited  to  the  operator’s  MDS  and  any 
shared  major  components  such  as  a  common  engine  or  munition.  A  key  characteristic  for 
upgrading  this  library  is  the  automated  push  of  information  to  the  user  when  certain 
criteria  are  met.  For  example,  an  operator  may  only  be  interested  in  deficiencies  related 
to  mission  tasks  for  the  MQ-1.  An  automated  email  could  alert  that  individual  operator 
when  a  DR  with  those  criteria  changes  status.  This  characteristic  is  also  useful  for 
current  users  of  JDRS.  For  example,  a  contractor  may  only  need  to  know  about  DRs 
related  to  a  particular  sub-system.  Rather  than  requiring  manually  checking  for  updated 
reports,  an  automated  system  would  save  the  contractor  significant  time  and  energy. 

The  final  recommendation  is  to  incorporate  OpenProposal,  or  a  similar  tool  based 
on  its  strengths,  for  the  Cockpit  Working  Group.  The  CWG’s  intent  is  to  evaluate  and 
improve  the  operational  suitability  and  effectiveness  of  an  aircraft’s  cockpit  or  remote 
operator  station  (AFI  63-112,  2011  :para  2. 1).  Likewise,  OpenProposal ’s  main  intent  is  to 
capture  desired  interface  changes  then  facilitate  a  meaningful  discussion  between 
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stakeholders  about  the  proposed  change  (Rashid,  2007:372).  The  electronic  discussion 
via  OpenProsoal  potentially  increases  the  frequency  of  communication,  decreases  the 
personal  effort  to  submit  feedback,  mitigates  the  geographic  dispersion  of  the  CWG 
members,  or  increases  the  maximum  amount  of  involved  operators. 

Potential  Future  Research 

There  are  a  few  recommendations  for  future  research  based  on  some  findings  and 
limitations  of  this  thesis.  The  limitations  revolve  around  perspectives  that  were  either  de- 
scoped  or  unavailable.  Other  areas  of  future  research  relate  to  answers  to  open-ended 
questions  that  were  tangential  to  this  study  but  could  improve  the  acquisition  system  in 
other  areas. 

Limitations 

This  study  had  significant  coverage  for  the  operator,  TES,  and  staff  perspectives. 
Other  perspectives  were  either  not  covered  or  were  not  covered  to  the  same  depth.  The 
first  lacking  perspective  is  the  Weapon  School.  Two  weapon  school  graduates  accepted 
the  study  invitation  and  while  both  responded  to  the  first  survey,  only  one  responded  to 
the  second.  The  reduced  amount  of  representation  is  significant  as  WPS  graduates  are 
more  likely  to  have  experienced  the  true  upper  bounds  on  an  MDS’s  mission 
effectiveness  and  suitability.  Additionally,  WPS  graduates  tend  to  populate  leadership 
positions,  therefore  the  WPS  has  a  concentration  of  influence  compared  to  the  Air  Force 
writ  large.  Future  research  with  WPS  graduates  could  investigate  whether  removing 
materiel  feedback  from  WEPTACs  negatively  impacted  WPS  graduates  ability  to  affect 
meaningful  changes  to  airframes.  Research  in  this  area  should  determine  if  current 
systems  are  adequate  to  capture  the  concerns  of  this  key  demographic. 
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Another  perspective  was  the  SPO:  two  SPO  members  accepted  the  invitation,  but 
only  one  responded  to  the  first  survey  and  neither  responded  to  the  second.  Additionally 
a  line  unit’s  SPO  liaison  was  not  available  for  the  second  survey.  The  SPO  is  significant 
as  a  system’s  Program  Manager  is  responsible  for  large  portions  of  the  acquisition 
process  including  significant  roles  in  both  the  DR  program  and  the  CWG.  Systemic 
changes  to  the  acquisition  process  require  understanding  the  role  of  the  SPO.  Specific 
research  could  determine  if  operator  feedback  from  current  programs  is  adequate  to 
properly  communicate  the  needs  of  the  community.  If  areas  of  weakness  exist,  research 
could  also  determine  solutions  to  increase  the  quality  of  the  feedback. 

The  study  was  deliberately  scoped  to  only  conventional  staff  members — for  RPA 
aircraft  this  was  Air  Combat  Command.  This  scope  focused  the  panel  towards  the 
processes  of  conventional  acquisition  through  the  lead  command  rather  than  special  or 
unconventional  acquisition.  Those  other  organizations  have  access  to  separate  funding 
which  further  complicates  an  already  complicated  subject  area.  With  initial  research 
complete,  an  expanded  research  effort  should  include  the  other  organizations.  Specific 
research  questions  could  detennine  if  unconventional  units  use  additional  undocumented 
methods  for  feedback  and,  if  they  exist,  should  those  methods  be  applied  to  conventional 
forces  as  well. 

The  last  limitation  has  less  impact  than  the  three  above:  the  focus  on  RPA  aircraft 
compared  to  manned  aircraft.  This  limitation  is  less  significant  as  the  RPA  community  is 
less  mature  than  most  manned  aircraft.  The  current  construct  for  RPA  is  merely  20  years 
old,  starting  in  April  of  1996  when  the  Secretary  of  Defense  selected  the  Air  Force  to 
operate  the  RQ-1  Predator  (“MQ-1B  Fact  Sheet,”  2010).  Most  other  communities — 
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fighters,  bombers,  airlift,  refueling,  etc — have  significantly  more  experience.  Additional 
studies  could  detennine  if  different  feedback  program  characteristics  are  desirable  for 
other  communities.  The  research  could  analyze  differences  based  on  airframe  maturity, 
mission  maturity,  or  tactical  vice  non-tactical  airframes. 

These  limitations  should  be  further  investigated  to  ensure  complete  understanding 
of  how  to  collect  user  feedback  from  a  variety  of  communities.  Without  additional 
research,  leadership  for  distinct  communities  may  not  be  applying  the  most  effective 
feedback  tools.  Properly  defined  requirements  may  assist  in  an  overall  improvement  of 
the  acquisitions  process. 

Tangential  Concerns  From  the  Panel 

A  significant  concern  from  Panelist  Hotel,  a  staff  member,  was  the  lack  of 
availability  for  experts  to  advise  other  stakeholders  on  the  requirements.  He  stated  the 
true  experts  were  not  typically  available  and  the  designated  representative  is  not  always 
an  expert  in  the  system.  The  lack  of  expertise  is  potentially  due  to  leadership  viewing  an 
assignment  as  liaison  as  less  prestigious  than  competing  assignments  for  career 
progression.  Additional  research  could  determine  if  that  condition  exists,  and  if  so,  how 
the  condition  should  change  to  ensure  staff  agencies  have  proper  operator  representation. 

Another  major  concern  from  the  panel  is  the  overall  timeliness  of  the  acquisition 
system.  Just  over  half  (7  of  13)  of  first  survey  respondents  stated  that  “timely”  is  desired 
characteristic  for  feedback  systems.  This  desire  is  likely  to  also  apply  to  the  entire 
acquisition  system.  A  study  dedicated  to  determining  a  prioritized  list  of  characteristics 
for  the  entire  acquisition  system  from  the  operator  viewpoint  could  determine  what 
aspects  of  the  acquisition  system  have  the  most  impact  to  the  operator.  If  timeliness  is 
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confirmed  as  a  broadly  desired  characteristic,  additional  studies  can  further  identify 
causes  and  fixes  for  the  untimely  portions  of  the  acquisition  process. 

Summary 

The  chapter  above  reviewed  the  current  feedback  systems  as  compared  to  the 
characteristics  the  panel  confirmed  as  significant.  This  comparison  revealed  that 
Deficiency  Reporting  meets  many  of  the  characteristics  but  could  be  improved  to  meet  all 
of  them.  Additionally,  the  OpenProposal  tool  also  most  of  the  desired  characteristics  and 
might  improve  the  CWG  if  implemented  in  that  program.  Finally,  recommended  future 
research  focused  on  other  acquisition  concerns  from  the  panel  and  perspectives  not 
covered  in  the  panel. 
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Appendix  A:  Glossary 


Acronyms 

ACC 

AETC 

AF 

AFFSA 

AFI 

AFRL 

AFSAS 

AFTO 

AI 

ARC 

ASRC 

CAF 

CAS 

CGA 

COCOM 

CONPLANS 

CSAR 

CWG 

DARPA 

DoD 

DR 

DT&E 

FFRDC 

GCS 

GPS 

GUI 

HAF 

HQ 

JCIDS 

JDRS 

JLLP 

JLLIS 

JUON 

LRE 

MAJCOM 

MCE 

MDS 


Air  Combat  Command 

Air  Education  and  Training  Command 

Air  Force  or  Air  Force  (Fonn) 

Air  Force  Flight  Standards  Agency 

Air  Force  Instruction 

Air  Force  Research  Labs 

Air  Force  Safety  Automated  System 

Air  Force  Technical  Order 

Air  Interdiction 

Air  Reserve  Component 

Air  Systems  Requirements  Council 

Combat  Air  Forces 

Close  Air  Support 

Capability  Gap  Assessment 

Combatant  Commands 

Concept  Plans 

Combat  Search  and  Rescue 

Cockpit  Working  Group 

Defense  Advanced  Research  Projects  Agency 

Department  of  Defense 

Deficiency  Report 

Developmental  Test  and  Evaluation 

Federally  Funded  Research  and  Development  Center 

Ground  Control  Station 

Global  Positioning  System 

Graphical  User  Interface 

Headquarters  Air  Force 

Headquarters 

Joint  Capabilities  Integration  and  Development  System 

Joint  Deficiency  Reporting  System 

Joint  Lessons  Learned  Program 

Joint  Lessons  Learned  Infonnation  System 

Joint  Urgent  Operational  Needs 

Launch  and  Recovery  Element 

Major  Command 

Mission  Control  Element 

Major  Design  Series 
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NASA 

NAF 

OPCON 

OPLANS 

OPR 

OT&E 

PACAF 

POC 

RPA 

SAB 

SCAR 

SPO 

SRD 

SWEBOK 

TES 

TIP 

TO 

TRB 

TST 

TSgt 

USAFE 

UON 

WEPTAC 

WEZ 

WPS 


National  Air  and  Space  Administration 
Numbered  Air  Force 
Operational  Control 
Operation  Plans 

Office  of  Primary  Responsibility 

Operational  Test  and  Evaluation 

Pacific  Air  Force 

Point  of  Contact 

Remotely  Piloted  Aircraft 

Scientific  Advisory  Board 

Strike  Coordination  and  Reconnaissance 

System  Program  Office 

System  Requirements  Document 

Software  Engineering  Body  of  Knowledge 

Test  and  Evaluation  Squadron 

Tactics  Improvement  Proposal 

Technical  Order 

Training  Review  Board 

Time-Sensitive  Target 

Technical  Sergeant 

United  States  Air  Forces  Europe 

Urgent  Operational  Need 

Weapons  and  Tactics  Conference 

Weapons  Engagement  Zone 

Weapons  School  (Squadron) 


Terms 


Ground  Control  Station  (GCS)  The  GCS  is  the  cockpit  for  a  remotely  piloted 

aircraft. 

Launch  &  Recovery  Element  (LRE)  Current  technology  allows  crews  to  control 

remotely  piloted  aircraft  from  any  global  location 
for  all  phases  of  flight  except  for  ground  operations, 
takeoff,  and  landing.  The  LRE  crew  links  to  RPA 
aircraft  using  line-of-sight  transmitters  for  the 
express  purpose  of  launch  and  recovery.  LRE 
crews  must  operate  from  the  same  airfield  as  the 
aircraft.  See  Mission  Control  Element. 


Mission  Control  Element  (MCE)  Current  technology  allows  crews  to  control 

remotely  piloted  aircraft  from  any  global  location 
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for  most  phases  of  flight.  MCE  crews  fly  aircraft 
via  satellite  link  to  complete  the  assigned  mission. 
MCE  crews  may  be  stationed  at  any  location  with 
enough  communication  architecture.  See  Launch 
and  Recovery  Element. 

Remotely  Piloted  Aircraft  (RPA)  One  of  many  terms  used  to  describe  an  aircraft 

whose  crew  is  not  inside  the  aircraft.  It  is  subset  of 
Unmanned  Aerial  Vehicles  to  specify  constant  man- 
in-the-loop  architecture.  Colloquially  it  describes 
MQ- 1  and  MQ-9  aircraft. 

Weapons  School  (WPS)  The  Air  Force  Weapons  School  is  a  6-month 

graduate-level  course  in  tactically  executing 
airpower.  The  Weapons  School  is  regarded  as  the 
pinnacle  of  tactical  prowess  and  is  often  the  hub  of 
emerging  tactics  for  the  Combat  Air  Force. 
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Appendix  B:  Initial  Questionnaire 


Delphi  Study  on  User  Feedback  Mechanisms  For  Materiel  Solutions 

Thank  you  for  participating  in  this  research.  I  appreciate  your  time  and  candid  responses.  The 
purpose  of  this  effort  is  to  define  the  characteristics  of  an  effective  user  feedback  system  to  modify 
materiel  solutions,  if  such  a  system  exists.  The  research  will  take  place  in  several  rounds,  with  each 
of  the  participants  provided  an  opportunity  to  respond  to  a  series  of  questions.  After  each  round,  the 
researcher  will  aggregate  individual  responses  into  a  coherent  whole  and  then  send  out  to  the  group  a 
refined  series  of  questions  and  an  instrument  to  assess  the  group  responses  from  the  previous  round. 
The  end  goal  is  to  reach  consensus  on  the  group’s  assessment  of  the  topic. 

1 .  Please  complete  this  instrument  and  return  it  electronically  no  later  than  15  May  2015. 

2.  There  are  8  questions.  This  questionnaire  is  “non-attribution,”  so  please  elaborate  fully  on  your 
answers.  Please  do  not  collaborate  with  other  individuals  in  providing  your  answers.  Once  all 
responses  are  received,  you  will  be  given  the  opportunity  to  revise  your  initial  responses  to 
questions  in  part  2.  Subsequent  rounds  will  be  announced  as  needed  and  all  research  will 
conclude  by  1  August  2015. 

Part  1 :  Basic  Demographics 

1 .  Personal  information 

a.  Level  of  education 

b.  Current  job  title 

c.  Most  recent  qualification 

2.  If  applicable,  how  many  total  years  have  you  flow  n  in  an  operational  flying  squadron  (BMC 
or  CMR  status)  and  with  which  group  (27  SOG,  49  OG,  432AEG,  732AEG,  etc)? 

3.  In  w'hat  capacities  have  you  dealt  with  the  acquisition  system? 

Part  2:  This  research  pertains  to  the  definition  of  an  effective  feedback  system  for  users  to  address 
materiel  chanues  (software  or  hardware).  Please  answer  and  elaborate  on  the  following: 

II' responses  include  FOUO  information  please  mark  and  transmit  appropriately. 

4.  Imagine  that  an  Air  Force  system  or  Major  Design  Series  (MDS)  has  recently  reached  IOC 
and  is  now  used  operationally.  What  stakeholders  (individual  job  positions  or  communities 
of  people)  should  have  power  to  change  the  system  to  be  more  effective  in  future  operations 
and  why?  Please  list  in  priority  order  and  include  any  discussion  or  justification  you  feel 
necessary. 

5.  What  is  your  personal  view  on  the  proper  role  of  end-users  or  line  operators  in  the  system 
modification  or  upgrade  process? 

6.  If  you  could  change  one  thing  about  the  methods  or  process  used  to  determine  a  system's 
requirements  inside  the  DoD  what  would  it  be? 

7.  What  are  your  top  2  or  3  preferred  characteristics  of  any  feedback  system? 

8.  What  type  of  information  should  any  feedback  system  seek  to  capture? 


86 


Please  note  the  following: 

1 .  Benefits  and  risks:  There  are  no  personal  benefits  or  risks  for  participating  in  this 
research.  Your  participation  should  take  less  than  30  minutes  per  round. 

2.  Voluntary  consent:  Your  participation  is  completely  voluntary.  You  have  the  right  to 
decline  to  answer  any  question,  as  well  as  refuse  to  participate  in  this  study  or  to 
withdraw  at  any  time.  Your  decision  of  whether  or  not  to  participate  will  not  result  in 
any  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Completion  of  the 
questionnaire  implies  your  consent  to  participate. 

3.  Confidentiality:  Your  responses  are  completely  confidential,  and  your  identity  will  only 
be  used  by  the  researchers  during  the  data  gathering  and  interpretation  phase  of  the 
research.  No  individual  data  will  be  reported;  only  data  in  aggregate  will  be  made  public. 
Individual  data  will  safeguarded  under  AFI  33-332  rules  for  FOUO  data  regardless  of 
whether  FOUO  data  is  actually  recorded.  If  you  have  any  questions  or  concerns  about 
your  participation  in  this  study,  please  contact: 

BRENT  T.  LANGHALS,  Lt  Col,  Ph.D. 

Assistant  Professor  of  Systems  Engineering 
Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Wright-Patterson  AFB,  OH 
Voice:  937-255-3636  (785-3636  DSN)  ext  7402 


Privacy  Act  of  1974  and  AFI  33-332 

The  Material  /  Information  contained  herein  falls  within  the  purview  of  the  Privacy  Act  of  1 974  and 
will  be  safeguarded  in  accordance  with  the  applicable  system  of  records  notice  and  AFI  33-332.  This 
study  is  anonymous.  No  attempt  to  identify  you  or  your  organization  will  be  made  unless  information 
indicates  a  credible  or  potential  threat.  By  participating  in  this  research,  you  acknowledge  that  the 
information  you  provide,  including  the  open  text  comments,  may  be  viewed  and  released  in 
accordance  with  the  Freedom  of  Information  Act.  Do  not  include  personal  identifying  information. 


Operational  Security  (OPSEC),  AFI  10-701 

Do  not  provide  OPSEC  information.  OPSEC  is  a  process  of  identifying,  analyzing  and  controlling 
critical  infonnation  indicating  friendly  actions  associated  with  military  operations  and  other  activities 
such  as:  1 )  Identify  those  actions  that  can  be  observed  by  adversary  intelligence  systems;  2) 
Determine  what  specific  indications  could  be  collected,  analyzed,  and  interpreted  to  derive  critical 
information  in  time  to  be  useful  to  adversaries;  and  3)  Select  and  execute  measures  that  eliminate  or 
reduce  to  an  acceptable  level  the  vulnerabilities  of  friendly  actions  to  adversary  exploitation.  Comply 
with  all  OPSEC  measures  outlined  in  AFI  10-701.  Do  not  provide  critical  infonnation  or  indicators. 
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Please  respond  to  this  request  for  your  assessment  electronically  and  return  it  to: 
ioseph.goldsmith@us.af.mil.  If  you  have  questions  on  any  of  this,  I  can  also  be  reached  at  (575)  572- 
8128  (work),  572-8128  (DSN),  or  702-862-0718  (cell).  Written  correspondence  can  be  addressed  to: 

Maj  Joseph  Goldsmith 
490  l5'  St 
Bldg  29,  Rm  1202 
Holloman  AFB,  NM  88330 

YOU  ARF.  MAKING  A  DECISION  WHETHER  OR  NOT  TO  PARTICIPATE.  YOUR 
SIGNATURE  INDICATES  THAT  YOU  HAVE  DECIDED  TO  PARTICIPATE  HAVING  READ 
THE  INFORMATION  PROVIDED  ABOVE. 


Volunteer  Signature. 


Date 


Volunteer  Name  (printed). 
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Appendix  C:  Second  and  Final  Questionnaire 

Delphi  Study  on  User  Feedback  Mechanisms  For  Materiel  Solutions  -  Survey  2 

Thank  you  for  participating  in  this  research.  1  appreciate  your  time  and  candid  responses.  The 
purpose  of  this  effort  is  to  define  the  characteristics  of  an  effective  user  feedback  system  to 
modify  materiel  solutions,  if  such  a  system  exists.  The  research  will  take  place  over  a  few 
rounds,  with  each  of  the  participants  provided  an  opportunity  to  respond  to  a  series  of  questions. 
After  each  round,  the  researcher  will  aggregate  individual  responses  into  a  coherent  whole  and 
then  send  out  to  the  group  a  refined  series  of  questions  and  an  instrument  to  assess  the  group 
responses  from  the  previous  round.  The  end  goal  is  to  reach  consensus  on  the  group’s 
assessment  of  the  topic. 

1 .  Please  complete  this  instrument  and  return  it  electronically  no  later  than  26  June  2015. 

2.  There  are  5  questions.  This  questionnaire  is  “non-attribution."  so  please  elaborate  fully  on 
your  answers.  Please  do  not  collaborate  with  other  individuals  in  providing  your  answers. 
Subsequent  rounds  will  be  announced  as  needed  and  all  research  will  conclude  by  1  August 
2015.  You  will  be  notified  when  the  research  is  terminated. 

Questions: 

].  User  involvement 

Virtually  all  surveys  indicated  users  should  be  involved  in  change  requests  to  some  degree.  What 
user  types  of  user  involvement  should  be  accepted? 

Check  all  that  apply: 

]  [Unfiltered.  unsolicited  user  feedback 

I  |  Unsolicited  user  feedback,  vetted  only  through  commanders  (SQ/cc  or  higher) 

I  |  Unsolicited  user  feedback,  vetted  through  a  functional  chain  (i.e.  not  commanders) 

]  [User  feedback  through  formal  studies  or  currently  established  feedback  systems 

(such  as  Deficiency  Reports.  Joint  Lesson  Learned  Library.  Safety  Boards,  and  Form 
1067  submissions;  note:  WEPTACs  no  longer  accept  materiel  feedback) 

I  |  Other _ 

2.  Final  determination 

Who  should  be  the  final  approval  authority  for  change  requests?  Assume  the  commander  in 
question  may  delegate  authority  to  a  lower  staff  member  for  minor  change  requests. 

-Q.1  .  The  lead  MAJCOM 

02  .  The  COCOM  that  has  OPCON  over  the  most  number  of  a  particular  MDS 
03  .  By  council  vote  (mark  the  appropriate  voting  members): 

□  The  lead  MAJCOM 
. _.Any  MAJCOM  that  owns  any  assets 

The  COCOM  with  the  most  OPCON  d  assets 
Any  COCOM  with  the  asset  in  a  war  plan 
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Many  responses  indicated  that  decisions  should  not  be  made  in  a  vacuum  and  other  stakeholders 
should  be  able  to  influence  the  final  decision  on  change  requests.  Who  of  the  following  should 
have  influence  or  give  suggestions  about  the  change  request?  Check  all  that  apply. 

Headquarters  AF 

□  The  lead  MAJC'OM 

□  Any  MAJCOM  that  owns  any  assets 

The  COCOM  with  the  most  OPCON'd  assets 
□Any  COCOM  with  the  asset  in  a  war  plan 
.□Applicable  weapons  school  commander 

□  Applicable  TES  commander 
.□Any  commander  (Sq/cc  or  higher) 

3.  Vetting  Process 

Several  responses  indicated  the  need  to  filter  feedback.  The  most  common  Air  Force  method  is 
to  have  a  functional  chain  of  command  sequentially  review  submissions.  For  example,  the  Form 
847  is  sequentially  processed  from  the  user  to  Squadron  Standardization/Evaluation  to  Group 
Stan/Eval  to  NAF  Stan/Eval  to  MAJCOM  Stan/Eval.  In  contrast,  a  prominent  commercial 
feedback  model  uses  a  group  discussion  between  the  user,  requirements  analysts,  and  engineers 
to  fully  capture  the  requirement. 

Which  of  the  following  proposed  methods  would  BEST  facilitate  timely  and  complete  reviews  of 
submissions? 

Q-Sequential  flow:  User-> Squadron  Weapons  and  Tactics->Group  Weapons  and 
Tactics->dccision  maker 

Q  Sequential  flow:  User- >  Squadron  Weapons  and  Tactics-^  Group  Weapons  and  Tactics 
-^Operational  Test&Eval  Squadron->dccision  maker 
^□Sequential  flow:  User->  Operational  Test&Eval  Sq uadron -^decision  maker 
(note:  this  flow  is  similar  to  the  current  construct  for  Deficiency  Reports) 

O  Sequential  flow:  Other _ 

Q  Group  discussion  with  user,  SPO,  decision  maker’s  staff 
O  Group  discussion  with  user,  TES,  decision  maker’s  staff 
Q  Group  discussion  with _ 

4.  Non-functional  characteristics 

About  half  of  the  responses  indicated  the  feedback  program  should  have  good  communication. 
Which  of  the  following  common  architectures  of  data  repositories  would  BEST  assist  good 
communication  without  sacrificing  timeliness  or  inducing  waste  by  excess  communication? 
,Q_An  open-access  online  library  where  interested  individuals  can  seek  out  and  read  submitted 
change  requests;  i.e.  a  pull  system  where  the  individual  must  pull  the  data  they  want. 

£lAn  open-access  online  library  with  automated  notifications  where  individuals  can  both  seek 
out  submitted  change  requests  and  be  notified  when  change  requests  meeting  certain  criteria 
change  status;  i.e.  a  push  and  a  pull  system  where  the  system  will  push  data  or  the  individual 
can  pull  the  data. 

iDAI  imited-access  online  library  where  only  trusted  agents  (i.e.  those  people  involved  in 

processing  the  change  requests)  have  access.  Other  individuals,  such  as  a  line  operator,  must 
request  the  data  from  the  trusted  agent. 

O  Other _ " 
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5.  Specific  feedback  items 


If  a  feedback  program  were  to  exist,  it  should  effectively  communicate  the  change  request  and 
the  complete  context  around  the  request.  Rank  order  the  following  potential  items  based  on  their 
ability  to  support  communication. 

13  Phase  of  (light  (e.g.  launch/recovery,  cruise,  recon  mission,  strike  mission,  etc) 

13  _Geographic  area(s)  with  the  most  impact  from  this  change  request 
13  MDS  sub-system  under  request 
13  Proposed  solution 

13  Attachments  to  support  videos,  pictures,  or  other  non-textual  feedback 
13  Jf  applicable,  acceptable  short-term  fix  or  alternate  solutions 
13  If  applicable,  estimated  urgency  (i.e.  time  before  the  request  is  no  longer  needed  / 
overcome  by  events) 

13  Justification:  i.e.  impact  to  the  crew  or  mission  if  not  implemented 

13 _  IJser  assessed  negative  aspects  to  the  change  request 

13  User  assessed  feasibility  analysis 

.13 _ _User  information  (name,  rank,  crew  position,  qualification,  contact  details) 

13  Reviewer  comments 

13  .Reviewer  information  (name,  rank,  crew  position,  qualification,  contact  details) 


6.  Open  Comments 

If  you  have  any  amplifying  comments  or  justifications  for  ANY  of  the  above  questions,  please 
add  them  here. 


If  responses  include  FOLIO  information  please  mark  and  transmit  appropriately. 

Please  note  the  following: 

1 .  Benefits  and  risks:  There  arc  no  personal  benefits  or  risks  for  participating  in  this 
research.  Your  participation  should  take  less  than  30  minutes  per  round. 

2.  Voluntary  consent:  Your  participation  is  completely  voluntary.  You  have  the  right 
to  decline  to  answer  any  question,  as  well  as  refuse  to  participate  in  this  study  or  to 
withdraw  at  any  time.  Your  decision  of  whether  or  not  to  participate  will  not  result  in 
any  penalty  or  loss  of  benefits  to  which  you  are  otherwise  entitled.  Completion  of  the 
questionnaire  implies  your  consent  to  participate. 


91 


3.  Confidentiality:  Your  responses  are  completely  confidential ,  and  your  identity  will 
only  be  used  by  the  researchers  during  the  data  gathering  and  interpretation  phase  of 
the  research.  No  individual  data  will  be  reported;  only  data  in  aggregate  will  be  made 
public.  Individual  data  will  safeguarded  under  AFI  33-332  rules  for  FOUO  data 
regardless  of  whether  FOUO  data  is  actually  recorded.  If  you  have  any  questions  or 
concerns  about  your  participation  in  this  study,  please  contact: 

BRENT  T.  LANGHALS,  Lt  Col.  Ph.D. 

Assistant  Professor  of  Systems  Engineering 
Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Wright-Patterson  AFB.  OH 
Voice:  937-255-3636  (785-3636  DSN)  ext  7402 


Privacy  Act  of  1974  and  AFI  33-332 

The  Material  /  Information  contained  herein  falls  within  the  purview  of  the  Privacy  Act  of  1974 
and  will  be  safeguarded  in  accordance  with  the  applicable  system  of  records  notice  and  AFI  33- 
332.  This  study  is  anonymous.  No  attempt  to  identify  you  or  your  organization  will  be  made 
unless  information  indicates  a  credible  or  potential  threat.  By  participating  in  this  research,  you 
acknowledge  that  the  information  you  provide,  including  the  open  text  comments,  may  be  viewed 
and  released  in  accordance  with  the  Freedom  of  Information  Act.  Do  not  include  personal 
identifying  information. 


Operational  Security  (OPSEC),  AFI  10-701 

Do  not  provide  OPSEC  information.  OPSEC  is  a  process  of  identifying,  analyzing  and 
controlling  critical  information  indicating  friendly  actions  associated  with  military  operations  and 
other  activities  such  as:  1 )  Identify  those  actions  that  can  be  observed  by  adversary  intelligence 
systems;  2)  Determine  what  specific  indications  could  be  collected,  analyzed,  and  interpreted  to 
derive  critical  information  in  time  to  be  useful  to  adversaries;  and  3)  Select  and  execute  measures 
that  eliminate  or  reduce  to  an  acceptable  level  the  vulnerabilities  of  friendly  actions  to  adversary 
exploitation.  Comply  with  all  OPSEC  measures  outlined  in  AFI  10-701 .  Do  not  provide  critical 
information  or  indicators. 

Please  respond  to  this  request  for  your  assessment  electronically  and  return  it  to: 
ioseph. goldsmith®1  us.af.mil.  If  you  have  questions  on  any  of  this,  I  can  also  be  reached  at  (575) 
572-8 1 28  (work),  572-8 1 28  (DSN),  or  702-862-07 1 8  (cell).  Written  correspondence  can  be 
addressed  to: 

Maj  Joseph  Goldsmith 
490  Is'  St 
Bldg  29,  Rm  1202 
Holloman  AFB,  NM  88330 

Internal  Use  Field 
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Appendix  D:  Air  Force  Form  847 


ATTACH  I  EXTRACT 


RECOMMENDATION  FOR  CHANGE  OF  PUBLICATION 


6.  PUBLICATION  NAME 


9  PAGE  NUMBER 


10.  MAJOR/SUB  PARAGRAPH  TITLE/NUMBER  OR  FIGURE  NUMBER 


11.  ITEM  NUMBER 

12.  OPR  (for  instructions) 

13.  IS  SUPPORTING  DOCUMENTATION  ATTACHED 

14.  SERIES  AFFECTED  (for  flight  manuals) 

□  YES  |  |  NO 

|yes  I  | NO 

15  TEXT  OR  FIGURE  AS  PRESENTLY  READS  (List  what  is  considered  to  be  incorrect  or  missing): 


16.  CHANGE  TO  READ  (Describe  the  desired  change) 


17  RATIONALE  (Provide  reason  or  additional  comments  for  this  recommendation) 


18.  NAME/RANK  (Of originator) 

19.  SIGNATURE 

20  ORGANIZATION 

21.  DSN 

FAX 

22.  FULL  MAI  UNG  ADDRESS 

23.  E-MAIL 

AF  FORM  847,  20090922 
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TO: 

FROM :  (Full  Address  Including  Zip  Code  and  DSN) 

SECTION  1  Q CONCUR  |~|  CONCUR  WITH  INTENT  |~|  DO  NOT  CONCUR 

REMARKS 

DATE 

TYPED  NAME,  GRADE,  AND  TITLE 

SIGNATURE 

TO 

FROM 

SECTION  2  Q  CONCUR  Q  CONCURWITH  INTENT  |  |  DO  NOT  CONCUR 

REMARKS 

DATE 

TYPED  NAME,  GRADE,  AND  TITLE 

SIGNATURE 

TO 

FROM 

SECTION  3  Q  CONCUR  Q  CONCURWITH  INTENT  |  |  DO  NOT  CONCUR 

REMARKS 

LEAD  MAJCOM 

COPIES  FORWARDED  TO 

DATE 

TYPED  NAME,  GRADE,  AND  TITLE 

SIGNATURE 

T 0  (FMM/Final  Approval  A  uthority) 

FROM  ((Full  Address  Including  Zip  Code  and  DSN) 

SECTION  4 

Q  CONCUR  Q  FORWARDED  TO  FOR  REVIEW  AND/OR  ACTION 

□  CONCUR  WITH  INTENT  □  DO  NOT  CONCUR  (See  comments  below) 

REMARKS 

DATE 

APPROVAL/DISAPPROVAL  AUTHORITY  NAME,  GRADE  &  TITLE 

SIGNATURE 

AF  FORM  847,  20090922 
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Appendix  E:  Air  Force  Form  1067 


PAGE  1  OF 


MODIFICATION  PROPOSAL 

PART  1  -  REQUEST  FOR  ACTION 

1  DATE: 

1.  INITIATOR 


2.  INITIATOR'S  POC  ORGANIZATION 


3.  USING  COMMAND  HQ  POINT  OF  CONTACT 


5.  ORGANIZATION  CONTROL  NUMBER 


6.  OTHER  NUMBERS 


7  AFFECTED  CONFIGURED  ITEM/SYSTEM : 


A  MDS/TMS/CEIL/CPIN 


D.  SRD  CODE 


8.  PURPOSE  (State  the  need  or  deficiency  to  be  corrected.  Include  expected  results.) 


9.  IM  PACT  (Urgency  of  need  and  impact  if  not  satisfied.) 


10.  CONSTRAINTS/ASSUMPTIONS/PROPOSED  SOLUTIONS 


11.  ORGANIZATION  VALIDATION 

I  DATE  RECEIVED: 

Q  A.  PROPOSED  REQUESTISVALIDATED  AS  AN  ORGANIZATION  NEED/REQUIREMENT  WHICH  REQUIRES  ACTION. 

Q  B.  PROPOSED  REQUEST  IS  DISAPPROVED  AND  IS  NOT  AN  ORGANIZATION  NEED/REQUIREMENT  WHICH  REQUIRES  ACTION. 

□  C.  PROPOSED  REQUEST  IS  RETURNED  TO  SUBMITTER  FOR  ADDITIONAL  INFORMATION 

D.  DATE 

E  NAME,  GRADE,  TITLE,  and  DSN  (Type  or  Print) 

F  SIGNATURE 

AF  FORM  1067,  19991101,  V2 


PREVIOUS  EDITIONS  ARE  OBSOLETE. 
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PAGE  2  OF 


PART  II  -  USING  COMMAND  VALIDATION 

DATE  RECEIVED: 

12.  USING  COMMAND  VALIDATION 

IT]  A.  PROPOSED  REQUEST  IS  VALIDATED  AS  AN  ORGANIZATION  NEED/REQUIREMENT  WHICH  REQUIRES  ACTION. 

n  B  PROPOSED  REQUEST  IS  DISAPPROVED  AND  IS  NOT  AN  ORGANIZATION  NEED/REQUIREMENT  WHICH  REQUIRES  ACTION. 

□  C.  PROPOSED  REQUEST  IS  RETURNED  TO  SUBMITTER  FOR  ADDITIONAL  INFORMATION. 

n  D.  FORWARD  TO  LEAD  COMMAND 

1  E.  USING  COMMAND  CONTROL  NO. 

F.  DATE 

G.  NAME,  GRADE,  TITLE,  and  DSN  (Type  or  Print) 

H.  SIGNATURE 

PART  III  -  LEAD  COMMAND  VALIDATION 

DATE  RECEIVED: 

13.  LEAD  COMMAND  ACTION  OFFICER 

14.  THRU  (Optional  Routing) 

15.  SINGLE  MANAGER  OFFICE 

16  MODIFICATION  TYPE  □  T"'  □  ™  □  PERMANENT  (P)  \J  P(S)-SAFETY 

17.  LEAD  COMMAND  CONTROL  NO. 

1 8.  LEAD  COMMAND  REMARKS  (Identify  any  constraints  or  assumptions) 


19.  LEAD  COMMAND  VALIDATION 

LJ  A.  VALIDATED  REQUEST 

□  B.  DISAPPROVED 

20.  NAME,  GRADE,  TITLE,  AND  DSN  (Type  or  Print) 

21.  SIGNATURE 

22.  DATE 

PART  IV  -  SINGLE  MANAGER  REVIEW  AND  APPROVAL 

|  DATE  RECEIVED: 

23.  SM  ACTION  OFFICER 


24.  CENTER  CONTROL  NUMBERS 

25.  TOTAL  BP/EEIC: 

A.  CENTER  Ml P  NO: 

Type  Funds 

Amount 

Type  Funds 

Amount 

B.  ECPNO: 

C.  TCTONO: 

26.  NR  OF  CIS  AFFECTED: 


27.  TOTAL  KITS  NEEDED: 


28.  ALSO  AFFECTS: 


SUPPORT  EQUIP 


□  SPARES  Q  SOFTWARE  Q 


AIRCREW  TRAINING  Q]  TRAINING  DEVICESA/ISUAL  AIDS  (Maint)  Q  TECH  DATA 


OTHER 


(Identify) 


29.  KIT  OR  UNIT  COST  30.  TOTAL  COST 

1  I  USER 


31.  LEAD  TIME 
~1  DEPOT  I  I  BOTH~ 


32.  INSTALLATION  (Begin) 
I  I  OTHER  | 


(Completed) 


33.  LEVEL  OF  ACCOMPLISHMENT. 


34.  USER  WORK  HOURS 


35.  DEPOT  WORK  HOURS: 


36.  TOTAL  WORK  HOURS: 


37.  MANUFACTURER: 


38.  AIRCRAFT  BREAKOUT: 


39.  ENGINEERING  REVIEW  RECOMMENDATION^ 


LH  APPROVED  IZI  Dl  SAPPROVED  (See  attached  remarks) 

40.  NAME,  GRADE,  TITLE,  AND  DSN  (Type  or  Print) 

41.  SIGNATURE 

42.  DATE 

PART  V  -  LEAD  COMMAND  CERTIFICATION/APPROVAL 

n  TEMPORARY  MOD  APPROVED 

O  PERMANENT  MOD  APPROVED  (Proceed  to  Budgeting) 

1  1  MOD  DISAPPROVED 

n  MNSfORD  TO  BE  DEVELOPED 

43.  NAME,  GRADE,  TITLE,  AND  DSN  (Type  or  Print) 

44.  SIGNATURE 

45.  DATE 

AFFORM  1067,  19991101,  V2  (REVERSE) 
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Appendix  F:  Air  Force  Technical  Order  Form  22 


TECHNICAL  MANUAL  (TM)  CHANGE  RECOMMENDATION  AND  REPLY  LCN  OMB  NO. 

(Use  IAW Completion  Instructions  and  TO  00-5-1)  0704-0158 

1.  PIM  (or equivalent) 

ORGANIZATION 

NAME 

PHONE  INITIAL  SUBMIT  DATE 

Q  APPROVED  Q  DISAPPROVED 

E-MAIL 

Click  to  sign 

2.  MAJCOM  CCP  (After  Review,.  Return  to  PIM) 

ORGANIZATION 

NAME 

PHONE  REVIEW  DATE 

Q  APPROVED  Q  DISAPPROVED 

E-MAIL 

Click  to  sign 

3.  LEAD  COMMAND  CCP  (After  Review.  Return  to  PIM) 

ORGANIZATION 

NAME 

PHONE  REVIEW  DATE 

□  APPROVED  Q  DISAPPROVED 

E-MAIL 

Click  to  sign 

4.  TO  MANAGEMENT  ACTIVITY  (After  Receipt  Forward  to  Evaluator) 

ORGANIZATION 

NAME 

PHONE  RECEIPT  DATE 

E-MAIL 

Click  to  sign 

5.  LOCAL  CONTROL  NUMBER  (LCN)  6.  PRIORITY  (Check One)  7.  CHANGE  TYPE  (Check  One) 

^EMERGENCY  ^URGENT  [^ROUTINE  ^CORRECTION  |  [IMPROVEMENT 

8.  INITIATOR 

NAME 

RANK  PHONE  DATE 

E-MAIL 

Click  to  sign 

9.  INITIATOR  SUPERVISOR 

NAME 

RANK  PHONE  DATE 

E-MAIL 

Click  to  agn 

10.  PUBLICATION  NUMBER 

14.  WORK  PACKAGE/WORK  CARD  ID 

11.  BASIC  DATE 

15.  PAGE  NUMBER 

12.  CHANGE  NUMBER  13.  CHANGE  DATE 

16.  PARAGRAPH  NUMBER  17.  FIGURE/TABLE  NUMBER 

18.  SHORT  DESCRIPTION  OF  DEFICIENCY 

19.  DEFICIENCY 
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LCN: 


20.  RECOMM ENDED  TM  CHANGE 


21.  SAVINGS/YR-  DOLLARS 

22.  SAVING S/YR-MANHOURS 

23.  EVALUATOR  (After  evaluation,  forward  to  supervisor)  24.  EVALUATOR/SUPERVISOR  (After  review,  return  to  TO  Management  Activity) 


NAME 

RANK 

PHONE 

RECEIPT  DATE 

EVALUATION  DATE 

E-MAIL 

Click  to  sign 

Click  to  sign 

25.  DISPOSITION 

26.  DISPOSITION/REMARKS 

□  approved 

□  deferred 

□  ABEYANCE 

□  advisement 

□  duplicate 

□  disapproved 

VERIFICATION  REQUIRED  BY  Q  PERFORMANCE  DESK-TOP  ANALYSIS 

□  other 

27.  IDEA  BENEFITS  ARE  □[INTANGIBLE  [~~| TANGIBLE  -  AMOUNT 

28.  CONTINUATION 
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ROLE 

AFTO  22  ABBREVIATED  COMPLETION  INSTRUCTIONS* 

WUC/LCN 

WUC  or  LCN  if  applicable. 

INITIATOR 
(Block  8) 

•  Complete  blocks  6-7  and  10-20.  Complete  blocks  21,  22  and  27,  if  applicable. 

•  Complete  block  8  and  digitally  sign.  Forward  signed  form  and  any  required  attachments  to  supervisor 

Initiator  Supervisor 
(Block  9) 

•  Review  blocks  6-7, 10-22  and  27  for  validity,  accuracy  and  completeness.  Make  necessary  changes  and  enter  corresponding  comments  in  block  28. 

•  Complete  block  9,  and  digitally  sign 

•  Forward  signed  form  and  all  attachments  to  PIM  (or  equivalent). 

PIM  (or Equivalent) 

(Block  1) 

•  Review  blocks  6-7,  10-22  and  27  for  validity,  accuracy  aid  completeness.  Make  appropriate  changes  and  enter  corresponding  comments  in  block  28 

•  Enter  Local  Control  Number  in  block  5. 

•  Enter  organization  information  and  e-mail  address  (preferably  an  organizational  e-mail)  into  block  1,  2,  and  3. 

•  See  routing  information,  via  AFNET  at  https://cs3.eis.af.mil/sites/00-TO-00-59/default.aspx 

•  Enter  the  Initial  Submit  Date  and  digitally  sign  block  1 

•  Forward  signed  form,  and  all  attachments,  to  the  first  reviewer 

•  Enter  dates  of  subsequent  reviews  in  block  28. 

•  Forward  to  the  TO  M  anagement  A  ctivity  in  block  4. 

Note:  Followup  with  reviewers  if  RC  is  not  returned  w#hin  14  calendar  days  of  submission.  Follow  up  with  the  evaluator  if  a  disposition  is  not  received  within  48  hours  for  an 

MAJCOM  and  Lead 

•  Review  blocks  6-7,  10-22  and  27  for  validity,  accuracy  and  completeness.  Make  appropriate  changes  and  enter  comments  in  block  28 

Command  CCP  Reviewer  *  Complete  block2  or  3,  as  appropriate,  including  review  date.  Digitally  sign 


(Blocks  2  and  3) 

•  Returned  signed  form,  and  all  attachments,  to  PIM  (or  equivalent)  (block  1) 

TO  Management  Activity 
(Block  4) 

•  Complete  block  4  and  digitally  sign 

•  Forward  signed  form,  and  all  attachments,  to  evaluator  (block  23) 

•  Upon  return  of  completed  form  from  evaluator/evaluator  supervisor,  return  to  PIM 

Evaluator 
(Block  23) 

•  Enter  receipt  date  in  block  23 

•  Review  blocks  6,  7, 10-22,  and  27  for  validity,  accuracy  and  completeness.  Make  appropriate  changes  and  enter  corresponding  comments  in  block  28 

•  Change  type  (block  7)  will  not  be  changed  without  the  approval  of  the  submitting  MAJCOM  CCP 

•  Recommended  disposition  in  block  25 

•  Provide  appropriate  verification  and  disposition  remarks  in  block  26 

•  Complete  block  23,  inducing  entering  evaluation  date,  and  digitally  sign 

•  Forward  completed  form  and  all  attachments,  to  supervisor 

Evaluator  Supervisor 
(Block  24) 

•  Review  recommended  disposition,  complete  block  24  and  digitally  sign. 

•  This  authority  may  be  delegated  to  the  evaluator.  If  so  delegated,  document  in  block  28,  along  with  the  first  level  supervisor's  name  and  e-mail  address. 

•  Return  completed  form  to  TO  M anagement  Activity  (Block  4) 

FOR  AFTO  FORM  22  DETAILE D  COMPLETION  INSTRUCTIONS,  SEE  TO  00-5-1 
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